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ADVANCED  ON-THE-JOB  TRAINING  SYSTEM  (AOTS) 
SYSTEM  LEVEL  TEST  &  EVALUATION  EXECUTIVE  SUMMARY 


Problems  with  the  Air  Force  On-the-Job  Training  (OJT)  system  began  surfacing  in 
the  late  1960s.  In  an  effort  to  define  specific  deficiencies,  the  Air  Staff  directed  studies  be 
conducted  by  the  Air  Force  Human  Resources  Laboratory  (Stephenson,  R.  W.,  and  Bur¬ 
kett,  J.  R.,  December  1975)  and  the  Air  Force  Inspector  General  (PN  76-269,  April  1977). 
Independent  studies  by  these  two  agencies  resulted  in  nearly  identical  findings: 

a.  Methodologies  for  identifying  specific  performance  and  training 
requirements  were  inadequate; 

b.  Valid  evaluations  of  task  knowledge  and  performance  were  infrequent 
and,  when  accomplished,  lacked  standardization; 

c.  Management  focus  was  on  Air  Force  Specialty  Code  (AFSC)  skill  level 
upgrade  training  and  the  ability  of  individuals  to  perform  duty 
position  tasks  was  being  not  addressed; 

d.  Reports  submitted  as  a  result  of  training  program  assessments  focused 
on  maintenance  of  training  records  rather  than  training 
effectiveness; 

e.  Methodologies  for  determining  capacities  of  operational  units  to 
conduct  training  were  lacking;  and 

f.  Personnel  assigned  to  the  training  specialty  were  being  utilized  to 
administer  rather  than  to  develop,  maintain  and  manage  training. 

The  Air  Staff  developed  and  implemented  several  initiatives  aimed  at  correcting  the 
deficiencies  identified  during  the  studies.  The  most  dynamic  initiative  was  the  design, 
development,  and  test  of  a  proof-of-concept,  prototype  computer-based  Advanced  On-the- 
job  Training  System  (AOTS)  that  included  all  functions  required  for  an  effective  training 
system.  The  purpose  for  the  prototype  AOTS  was  two-fold:  1)  to  determine  whether 
modern  computer  and  training  technologies  could  be  applied  to  provide  solutions  for  the 
deficiencies  identified  in  the  two  studies;  and  2)  to  determine  if  the  system  would  be  suit¬ 
able  for  implementation  across  the  Air  Force  operational  environment. 

The  Air  Staff  mandated  that  the  prototype  AOTS  be  designed,  developed,  and  tested 
in  an  operational  environment.  The  purpose  was  to  ensure  all  functional  requirements 
were  included  in  the  design  and  subjected  to  the  conditions  under  which  the  system  would 
operate  if  approved  for  Air  Force-wide  implementation.  Bergstrom  AFB,  Austin,  Texas 
was  selected  as  the  site  at  which  the  development  of  design  and  the  system  would  occur. 
Bergstrom  and  Ellington  ANGB,  Houston,  Texas  were  selected  as  the  sites  at  which  the 
System  Level  Test  &  Evaluation  (SLT&E)  of  the  prototype  AOTS  would  occur.  All  Air 
Force  components  (Active,  Guard  and  Reserves)  and  five  Air  Force  Specialties  were 
selected  to  participate  in  the  SLT&E.  The  selected  AFSs  were: 

a.  Aircraft  Maintenance,  AFSC  452X4D/M; 

b.  Aerospace  Propulsion,  AFSC  454X0A; 

c.  Personnel,  AFSC  732X0; 

d.  Security  Police  -  Security,  AFSC  811X0;  and 

e.  Security  Police  -  Law  Enforcement,  AFSC  811X2. 


The  prototype  AOTS  was  one  of  the  first  major  systems  other  than  a  weapon  related 
system  designated  to  use  Adapt  as  the  primary  programming  language.  In  addition,  the 
prototype  AOTS  undertaking  represented  the  most  comprehensive  single  Instructional  Sys¬ 
tems  Development  (ISD)  effort  attempted  to  date  in  the  Air  Force  OJT  environment. 

Four  years  were  allocated  for  the  AOTS  project.  The  effort  was  divided  into  three  phases: 
Phase  1  for  the  preliminary  design  of  the  system  and  subsystems;  Phase  2  for  the  detailed 
design  of  the  subsystems  and  components  and  for  the  development  of  software  required  to 
implement  the  system;  and  Phase  3  for  the  SLT&E. 

The  prototype  AOTS  was  comprised  of  five  subsystems:  Management,  Evaluation, 
Training  Development  and  Delivery,  Computer  Support,  and  Personnel  and  Support.  The 
Management  subsystem  provided  capabilities  to  identify  training  and  evaluation  require¬ 
ments,  identify  resources  required  for  training  and  evaluation,  schedule  training  and  evalu¬ 
ation  events,  manage  trainee  progress,  maintain  trainee  progress  data,  and  generate 
standard  and  ad  hoc  training  reports.  The  Evaluation  subsystem  provided  capabilities  to 
develop  and  maintain  behavioral  objectives,  develop  and  maintain  test  items,  construct 
tests,  deliver  and  score  tests,  collect  and  analyze  test  data,  and  assess  the  quality  of  training 
programs.  The  Training  Development  and  Delivery  subsystem  provided  capabilities  to 
develop  and  deliver  computer-assisted  instruction.  The  Computer  Support  subsystem 
implemented  system  requirements  and  provided  the  necessary  equipment.  The  Personnel 
and  Support  subsystem  supported  the  determination  of  logistics  support,  maintainability, 
reliability  and  human  factors  requirements. 

A  DigitalftM,  VAX  8650  computer  located  at  Brooks  AFB,  TX  hosted  the  system. 
During  the  periods  of  development  and  SLT&E,  dedicated  high-speed  digital  communica¬ 
tion  lines  were  used  to  transmit  data  between  the  host  computer  and  work  stations  at  the 
bases.  Infotrongw,  Multiplexers  were  used  to  control  and  distribute  communications  at  the 
bases.  A  typical  work  station  at  a  base  included  a  Zenith,™,  248  personal  computer,  a  Scan- 
tron<">  optical  mark  sense  reader  and  an  Alps,™,  P2000G  dot  matrix  printer. 

AFHRL  and  contractor  personnel  worked  closely  together  to  ensure  the  system 
design  met  all  contractual  and  functional  requirements.  Following  AFHRL  approval  of  the 
prototype  AOTS  design,  the  project  progressed  to  the  development  phase.  Software 
required  to  implement  the  design,  and  evaluation  and  instructional  materials  required  for 
use  during  task  training  of  personnel  participating  in  the  prototype  AOTS  SLT&E  were 
developed  during  this  phase. 

Prior  to  the  scheduled  SLT&E  start  date,  the  equipment  necessary  to  establish  proto¬ 
type  AOTS  work  stations  was  installed  in  the  work  centers  designated  to  participate  in  the 
SLT&E.  AOTS  implementation  was  accomplished  in  phases  with  no  major  problems. 
Immediately  following  implementation  within  a  work  center,  the  personnel  assigned  to  that 
work  center  were  trained  to  operate  the  system  and  the  conventional  method  of  OJT  was 
replaced  by  the  prototype  AOTS  method. 

A  multifacted  Master  Test  Plan  (MTP)  was  developed  to  outline  procedures  for 
determining  compliance,  performance,  acceptance  and  suitability  of  the  prototype  AOTS. 
Data  were  gathered  through  administering  surveys,  conducting  structured  interviews, 
reviewing  Air  Force  reports,  reviewing  training  records,  reviewing  results  of  knowledge  and 
performance  tests,  and  reviewing  reports  generated  by  the  AOTS.  Baseline  data  for  exper¬ 
imental  groups  were  collected  during  the  year  preceding  the  SLT&E.  Data  were  collected 
for  experimental  groups  and  control  groups  during  the  period  of  the  SLT&E.  Baseline 
data  and  data  for  control  groups  were  compared  with  data  for  experimental  groups. 


The  results  of  prototype  AOTS  SLT&E  revealed  the  following  concerning  com¬ 
pliance,  performance,  acceptance  and  suitability  of  the  AOTS: 

1.  Compliance.  The  prototype  AOTS  met  all  specified  functional  requirements 
and  established  system  performance  goals.  Terminal  and  data  search  response 
times  were  within  the  specified  time  limits.  External  interfaces  were  estab¬ 
lished.  Hardware  reliability  exceeded  established  goals.  Maintenance  goals 
were  met  or  exceeded,  with  minor  exceptions  occurring  only  when  users  failed 
to  report  problems  in  a  timely  manner.  The  availability  of  hardware  compo¬ 
nents  exceeded  the  established  goal. 

2.  Performance.  Indications  were  that  the  prototype  AOTS  performed  better 
than  the  conventional  OJT  system.  Training  managers  reported  that  less  time 
was  required  for  performing  training  management  functions  under  the  AOTS 
than  under  the  conventional  OJT  system.  Airmen  trained  under  the  AOTS 
scored  higher  on  task  related  tests  than  did  airmen  trained  under  the  conven¬ 
tional  system.  Managerial  personnel  indicated  that  airmen  trained  under 
AOTS  performed  tasks  better  and  became  position  qualified  sooner  than 
airman  trained  under  the  conventional  system.  These  same  individuals  also 
indicated  their  belief  that  the  system  improved  unit  effectiveness  and  would 
improve  combat  readiness  if  implemented  throughout  all  workcenters. 

3.  Acceptance.  Commanders  of  participating  units,  users  and  Major  Command 
(MAJCOM)  observers  of  the  prototype  AOTS  gave  the  system  high  accep¬ 
tance  ratings.  Users  favored  the  methods  for  managing  and  recording  training 
provided  by  the  AOTS  over  conventional  methods.  Users  indicated  that  the 
system  should  be  implemented  throughout  the  Air  Force  operational  environ¬ 
ment. 

4.  Suitability.  Indications  were  that  the  prototype  AOTS  could  overcome  defi¬ 
ciencies  identified  in  the  conventional  OJT  system.  Users  believed  the  system 
was  needed  to  enhance  training  within  their  specialties.  Unit  commanders 
believed  the  system  satisfied  their  unit  training  requirements,  would  signifi¬ 
cantly  improve  the  combat  capabilities  of  their  umts,  and  was  suitable  for  Air 
Force-wide  implementation.  Goals  to  reduce  the  amount  of  time  required  for 
performing  training  administrative  duties  and  providing  more  time  for  the  con¬ 
duct  of  training  were  met. 

Representatives  from  MAJCOMS  performed  an  assessment  of  the  prototype  AOTS. 
These  representatives  received  a  thorough  demonstration  of  the  system,  were  provided 
opportunities  to  interface  with  the  system,  and  observed  system  operations  in  the  partici¬ 
pating  work  centers.  Following  the  orientation,  the  majority  of  these  representatives  indi¬ 
cated  a  desire  to  implement  the  system  across  the  Air  Force. 

The  Air  Force  considered  the  AOTS  project  to  be  a  success.  The  system  was 
designed,  developed,  and  tested  on  schedule  and  within  budget.  The  AOTS  demonstrated 
that  modern  computer  and  training  technologies  could  be  successfully  applied  in  the  opera¬ 
tional  environment.  The  results  from  testing  the  system  were  positive,  ft  is  recommended 
that  the  Air  Force  proceed  with  efforts  to  implement  automated  OJT  capabilities  across 
the  operational  environment. 
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This  report  was  developed  by  Douglas  Aircraft  Company,  the  development  contractor,  under 
Government  Contract  F33615-C-84-0059.  The  AFHRL  Work  Unit  number  for  the  project  is 
2557-00-02.  The  primary  office  of  responsibility  for  management  of  the  contract  is  the  Air  Force 
Human  Resources  Laboratory,  Training  Systems  Division,  and  the  Air  Force  AOTS  manager  is 
Major  Jack  Blackhurst. 
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1  INTRODUCTION 


This  document  provides  a  summary  of  a  four  year,  three  phase  effort  to  design, 
develop  and  implement  into  operational  work  centers,  the  prototype  Advanced  On-the-Job 
Training  System  (AOTS).  Besides  being  one  of  the  first  major  systems  other  than  a 
weapon  system  to  use  Ada<*>  as  the  primary  programming  language,  the  prototype  AOTS 
represented  the  most  ambitious  and  comprehensive  Instructional  Systems  Development 
(ISD)  effort  attempted  to  date  within  the  Air  Force  OJT  environment. 

This  report  outlines  the  procedures  applied  to  evaluate  the  AOTS  and  reports  the 
results  of  a  multifaceted  field  evaluation  conducted  while  the  AOTS  was  implemented  in 
selected  Air  Force  work  centers. 

For  a  detailed  description  of  the  AOTS  design  and  development  effort,  see  the  Tech¬ 
nical  Report  for  the  Advanced  On-the-Job  Training  System  (AOTS)  dated  31  October 
1989.  For  a  description  of  the  intended  AOTS  SLT&E  data  collection  and  analysis 
procedures,  see  the  Master  Test  Plan  (MTP)  for  the  Advanced  On-the-Job  Training  System 
dated  10  October  1988.  For  an  explanation  of  how  the  AOTS  functioned  in  operational 
work  centers,  see  the  Operational  Guide  to  the  Prototype  Advanced  On-the-Job  Training 
System  dated  10  June  1988. 
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2  AOTS  BACKGROUND 


For  more  than  a  decade,  the  increasing  complexity  of  weapon  systems  and  equip¬ 
ment,  the  loss  of  qualified  middle  management  personnel  through  forced  manpower  reduc¬ 
tions,  and  increasing  mission  demands  have  made  On-the-Job  Training  (OJT)  increasingly 
more  difficult  to  conduct.  To  identify  specific  OJT  problems  and  viable  solutions  to  those 
problems,  the  Air  Staff  requested  that  the  Air  Force  Human  Resources  Laboratory 
(AFHRL)  conduct  a  study  of  the  Air  Force  OJT  system  (Stephenson  and  Burkett,  1975) 
and  that  the  Air  Force  Inspector  General  (IG)  conduct  a  Functional  Management  Inspec¬ 
tion  (FMJ)  of  the  OJT  system  (PN  76-269,  April  1977).  Based  on  the  findings  during  both 
the  AFHRL  study  and  the  Air  Force  IG  FMI  the  following  recommendations  were  made: 

a.  Develop  better  methods  for  identifying  and  updating  specific  performance  and 
training  requirements; 

b.  Perform  frequent,  valid  and  reliable  evaluations  of  job  task  performance; 

c.  Emphasize  the  quality  of  training  programs; 

d.  Emphasize  the  relevance  of  training  to  mission  accomplishments; 

e.  Deemphasize  training  administration; 

f.  Determine  the  cost  of  training  on  the  job; 

g.  Determine  the  capacity  of  operational  units  to  conduct  training  on  the  job;  and 

h.  Utilize  training  personnel  as  training  developers,  as  opposed  to  training 
administrators 

Following  the  AFHRL  study  and  the  Air  Force  IG  FMI,  the  Air  Staff  requested  that 
the  AFHRL  manage  an  initiative  to  design,  develop  and  test  a  prototype  computer-based 
system  for  Air  Force  OJT.  The  Air  Staff  specified  that  the  prototype  system  be  developed, 
tested,  and  evaluated  within  an  operational  environment  that  represented  typical  Air  Force 
work  centers.  Thus,  the  idea  of  an  OJT  system  designed  to  include  all  functions  of  an 
effective  training  system  was  conceived  and  named  the  Advanced  On-the-Job  Training  Sys¬ 
tem  (AOTS). 

The  primary  objective  of  the  prototype  AOTS  project  was  to  demonstrate  that, 
through  the  application  of  modern  computer  and  training  technologies,  deficiencies  found 
to  exist  in  conventional  on-the-job  traimng  (OJT)  could  be  corrected.  To  meet  the  mission 
objective,  AOTS  was  to  focus  on  job  task  proficiency  by  specifying  and  defining  the  tasks 
required  in  a  given  duty  position  and  by  defining  the  training  required  to  become  fully  posi¬ 
tion  qualified.  The  prototype  AOTS  was  to: 

a.  Provide  workable  and  cost  effective  methods  for  evaluating  task  performance 
in  an  operational  setting; 

b.  Ease  the  inherent  difficulties  of  using  operational  equipment  for  OJT; 

c.  Address  the  problem  of  the  limited  availability  of  technically  qualified  person¬ 
nel  who  could  act  as  trainers  and  the  limited  training  skills  of  those  personnel: 

d.  Ease  the  paperwork  burden  associated  with  scheduling  both  expected  and 
unique  OJT  opportunities,  tracking  trainee  training  progress,  evaluating  asso¬ 
ciated  task  knowledge  and  task  performance,  managing  the  flow  of  OJT  data; 
and 
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e.  Ultimately  increase  individual  and  unit  productivity  and  readiness. 

The  preliminary  design  of  the  training  system  and  subsystems  was  established  in 
Phase  1.  The  detailed  designs  of  the  subsystems  and  components  and  the  software 
required  to  implement  the  system  were  developed  in  Phase  2.  A  system  level  test  and 
evaluation  (SLT&E)  occurred  in  Phase  3  during  the  implementation  of  the  AOTS  in  an 
operational  environment.  Figure  2.1  displays  an  overview  of  the  AOTS  schedule.  The 
remainder  of  this  section  provides  a  brief  description  of  the  procedures  followed  during  the 
three  phases  of  the  AOTS  design,  development  and  implementation. 


1985 

1986 

1 

1987 

1  1 

1988 

1989 

1 

i 

AUG 

1 

15 

MAY 

1 

1  1 

31 

JUL 

1 

1 

31  31 

JUL  OCT 

PHASE  1 

PRELIMINARY 

DESIGN 

PHASE  2 

DETAILED  DESIGN  AND 
DEVELOPMENT 

1 

PHASE  3  WRITE^ 

SYSTEM  LEVEL  REPORTS 

TEST  AND 

EVALUATION 

Figure  2. 1 .  The  AOTS  Master  Schedule 
2.1  Phase  1  -  Preliminary  Design 

The  Air  Force  specified  that  the  prototype  AOTS  provide  state-of-the-art  capabili¬ 
ties.  Existing  technologies  were  considered  at  the  start  of  the  preliminary  design. 
Throughout  the  design,  emerging  technologies  were  scrutinized  to  determine  whether  they 
could  be  employed  to  enhance  the  prototype  AOTS. 


The  methodology  chosen  to  accomplish  the  Preliminary  Design  was  top-down  struc¬ 
tured  analysis  and  design.  Hierarchy  diagrams  and  Data  Flow  Diagrams  (DFDs)  were  used 
as  tools  to  describe  the  design.  The  hierarchy  diagrams  were  developed  to  depict  the 
structure  of  the  prototype  AOTS  from  system  level  through  subsystem,  components,  and 
functions.  These  diagrams  also  established  the  vehicle  for  system  naming  and  numbering 
(identification)  conventions.  DFDs  were  developed  to  show  processes  that  would  occur 
with  the  system,  the  internal  and  external  interface  requirements,  the  data  flowing  into  and 
from  processes  and  external  entities,  and  data  being  stored.  Each  of  these  DFD  entities 
were  described  in  data  dictionaries.  A  Computer-Aided  Software  Engineering  (CASE) 
tool,  DFDDraw<q,  was  used  as  an  aid  in  constructing  DFDs. 


Design  efforts  started  with  the  development  of  system  level,  subsystem,  and  compo¬ 
nent  diagrams.  Design  walkthroughs  were  conducted  and  attended  by  personnel  assigned 
to  the  project.  Each  diagram  was  discussed  in  detail  to  ensure  not  only  that  contractual 
and  functional  requirements  were  met,  but  also  that  the  design  would  satisfy  training 
requirements  for  all  AFSs  selected  to  participate  in  the  effort  and  the  corresponding  opera¬ 
tional  environments.  All  required  changes  were  accomplished,  followed  by  further  walk¬ 
throughs.  These  procedures  were  followed  until  all  personnel  concerned  were  satisfied 
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that  requirements  were  properly  addressed  in  the  design.  Preliminary  Design  Reviews 
(PDRs)  were  conducted  for  all  subsystems  and  components,  to  obtain  Air  Force  approval 
prior  to  proceeding  to  Phase  2. 


2.2  Phase  2  -  AOTS  Detail  Design  and  Development 

The  contract  for  the  design,  development,  test  and  evaluation  of  the  prototype  AOTS 
specified  that  all  AOTS  software  would  be  delivered  to  AFHRL  at  the  end  of  Phase  2. 
However,  automated  capabilities  were  needed  early  in  Phase  2  to  support  training  develop¬ 
ment  requirements.  A  two  part  development  effort  was  undertaken  to  provide  the  needed 
training  development  capabilities  earlier  in  the  phase. 

The  first  part  of  the  effort  was  performed  using  a  traditional  waterfall  approach.  The 
design  was  completed  for  a  portion  of  the  system  ana  presented  for  Air  Force  approval  at  a 
Critical  Design  Review  (CDR).  The  coding  and  informal  testing  started  after  this  appro¬ 
val,  followed  by  formal  testing  after  all  development  was  completed. 


During  the  second  part  of  the  development  effort  an  incremental  or  evolutionary 
approach  was  used.  Logically  related  portions  of  the  code  were  designed,  approved  via 
walkthroughs,  coded,  tested,  and  released.  Air  Force  personnel  participated  in  the  walk¬ 
throughs  and  in  the  formal  testing.  This  methodology  provided  an  ever  increasing  capabil¬ 
ity  to  AFHRL  with  earlier  user  feedback  into  the  development  process  improving  the  qual¬ 
ity  of  later  released  increments.  The  increments  were  developed  to  provide  the  most 
critical  capabilities  first. 

The  actual  design  methodology  was  the  same  in  both  parts  of  the  design  effort.  This 
involved  a  top-down  structured  design  with  a  heavy  object  oriented  influence  when  the 
design  was  translated  into  an  Adaw  package  structure.  The  design  was  presented  using  an 
Adaw  based  programming  design  language  that  facilitated  the  direct  transition  to  code. 

The  Ada<°)  compiler  was  used  to  validate  the  design  by  producing  executable  shells  to  check 
interfaces  from  both  a  data  and  flow  of  control  point  of  view.  The  design  was  then 
expanded  to  include  full  executable  code. 

AdaM  program  language  was  used  as  both  the  design  and  primary  implementation 
language.  The  Statistical  Analysis  System  (SAS)n  was  used  for  generating  standard  and  ad 
hoc  reports  from  the  prototype  AOTS  and  to  support  analyses  of  System  Level  Test  & 
Evaluation  (SLT&E)  data.  Assembly  language  code  was  used  to  make  the  Zenith(TM,248 
Personal  Computers  compatible  with  the  Digital^i  VAX  8650  computer. 


To  meet  the  functional  requirements,  the  prototype  AOTS  was  comprised  of  five  sub¬ 
systems;  Management,  Evaluation,  Training  Development  and  Delivery,  Computer  Sup¬ 
port,  and  Personnel  Support.  Figure  2.2  provides  a  brief  sketch  of  the  subsystems.  For  a 
complete  description  of  the  AOTS  subsystems,  see  each  respective  AOTS  subsystem  Prime 
Item  Specification. 
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MANAGEMENT 

SUBSYSTEM 


EVALUATION 

SUBSYSTEM 


TRAINING 

DEVELOPMENT 

AND 

DELIVERY 

SUBSYSTEM 


COMPUTER 

SUPPORT 

SUBSYSTEM 


PERSONNEL 

AND 

SUPPORT 

SUBSYSTEM 


•  DEVELOP  MASTER  TASK  LISTS  •  MANAGE  TRAINING  PROGRESS 

•  MAINTAIN  PERFORMANCE  AND  PROFICIENCY  DATA*  MANAGE  RESOURCE  INVENTORIES 

•  PRIORITIZE  DEVELOPMENT  REQUIREMENTS  •  SCHEDULE  EVENTS 

•  MANAGE  POSITION  TRAINING  REQUIREMENTS 

•  MANAGE  OTHER  TRAINING  REQUIREMENTS 

•  MANAGE  LOCAL  AND  NEW  TASKS 

•  GENERATE  TRAINING  RECORDS 

•  DIAGNOSE  TRAINING  REQUIREMENTS 


•  DEVELOP  OBJECTIVES 

•  PLAN  EVALUATION  MATERIALS 

•  MAINTAIN  TEST  ITEM  BANK 

•  OEVELOP  EVALUATION  MODULES 

•  DELIVER  EVALUATION  MATERIALS 

•  ANALYZE  TEST  ITEMS 

•  EVALUATE  TASK  KNOWLEDGE 

•  EVALUATE  TASK  PERFORMANCE 


•  CONTROL  ACCESS  TO  TESTS 

•  SCORE  TESTS 

•  COLLECT  TEST  DATA 

•  GENERATE  QUALITY  CONTROL  NOTICES 

•  GENERATE  QUALITY  CONTROL  EVENTS 

•  GENERATE  QUALITY  CONTROL 
FOLLOW-UP 


DEVELOP  INSTRUCTIONAL  MATERIALS 

validate  instructional  materials 

STORE  INSTRUCTIONAL  MATERIALS 
DELIVER  INSTRUCTIONAL  MATERIALS 
COLLECT  DELIVERY  DATA 


PR0VI0E  CENTRAL  COMPUTER  EQUIPMENT 
PROVIOE  WORKSTATION  EQUIPMENT 
PROVIOE  COMMUNICATIONS  EQUIPMENT 
PROVIDE  VIRTUAL  MACHINE  INTERFACE 
PROVIDE  APPLICATIONS  SUPPORT  ENVIRONMENT 
PROVIOE  OPERATING  SYSTEM 
PROVIDE  ADA  COMPILER 
PROVIOE  ADA  LINKER 


•  PROVIDE  ADA  DEBUGGER 

•  PROVIDE  SOFTWARE  EDITOR 

•  PROVIDE  AOTS  TERMINAL  EMULATOR 

•  CONTROL  SYSTEM  ACCESS 

•  IMPLEMENT  REQUIREMENTS 


•  DETERMINE  HARDWARE  REQUIREMENTS 

•  DETERMINE  SOFTWARE  MAINTENANCE  REQUIREMENTS 

•  DETERMINE  MAINTENANCE  REQUIREMENTS 

•  DETERMINE  RELIABILITY  REQUIREMENTS 

•  DETERMINE  HUMAN  FACTORS  REQUIREMENTS 

•  DETERMINE  SAFETY  REQUIREMENTS 

•  PREPARE  TRANSITION  PLAN 

•  PREPARE  EXPANSION  PLAN 


Figure  2.2.  AOTS  Subsystems 

2.3  Phase  3  -  AOTS  Implementation 

The  Air  Staff  sponsor  for  the  project,  Headquarters  United  States  Air  Force/Direc- 
tor  of  Personnel  Plans  and  Programs  (HQ  USAF/DPP)  directed  that  the  AOTS  prototype 
be  developed  and  evaluated  within  an  operational  environment  to  ensure  that  the  system 
would  be  useful  and  acceptable  to  Air  Force  users.  The  criteria  used  to  select  the  locations 
included: 


a.  a  representative  flying  mission; 

b.  total  force  units  (active  Air  Force,  Air  Force  Reserve,  and  Air  National 
Guard)  in  close  proximity; 
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c.  all  units  of  the  total  force  components  flying  the  same  weapon  system;  and 

d.  a  stable  weapon  system. 

HQ  USAF/DPP  selected  Bergstrom  AFB  and  Ellington  ANGB,  Texas  as  the  instal¬ 
lations  on  which  to  implement  the  AOTS.  The  main  participants  of  th?  AOTS  experiments 
were  Active,  Reserve,  and  Air  National  Guard  individuals  within  each  of  the  following  Air 
Force  Specialties; 

a.  Tactical  Aircraft  Maintenance,  AFSC  452X4D/M,  (previously  AFSC 
43 IX  1/C); 

b.  Aerospace  Propulsion,  AFSC  454X0A  (previously  AFSC  426X2); 

c.  Personnel,  AFSC  732X0;  and 

d.  Security  Police,  AFSCs  811X0/811X2. 

The  five  selected  specialties  represented  approximately  20  percent  of  the  total 
enlisted  US  Air  Force  and  a  wide  range  of  job  types  requiring  moderately  different  OJT 
methodologies.  The  Active,  Air  Force  Reserve,  and  Air  National  Guard  units  assigned  to 
the  AOTS  treatment  group  and  corresponding  control  group  are  listed  in  Table  2.1. 

AFHRL  implemented  the  prototype  AOTS  in  phases.  The  rationale  for  implement¬ 
ing  the  system  in  phases  included:  1)  initial  implementation  problems  could  be  resolved 
before  all  work  centers  were  affected,  2)  if  problems  occurred,  training  would  be  minimally 
impacted  since  the  majority  of  work  centers  would  still  be  operating  using  conventional 
methods,  3)  personnel  at  Ellington  ANGB  were  having  difficulty  with  the  timely  installa¬ 
tion  of  communication  lines,  and  4)  concern  that  the  work  centers  would  not  be  provided 
the  full  service  required  with  initial  implementation,  given  the  limited  number  of  personnel 
resources  assigned  to  the  project. 
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TABLE  2.1 

EXPERIMENTAL  AND  CONTROL*  UNITS  BY  COMPONENT 
COMPONENT  |  EXPERIMENTAL  GROUP  " 


Active 

67th  Aircraft  Generation  Squadron,  12th  Aircraft  Maintenance  Unit  Flightline  Section 

Active 

67th  Component  Repair  Squadron,  Jet  Engine  Intermediate  Maintenance  Section 

Active 

67th  Equipment  Maintenance  Squadron,  12th  &  62nd  Aircraft  Maintenance  Unit 
Inspection  Section 

Active 

67th  Mission  Support  Squadron,  Consolidated  Base  Personnel  Office  Base  On-the- 
Job  Training  Management  &  Quality  Force  Sections 

Active 

67th  Security  Police  Squadron,  All  Sections 

Active 

67th  Tactical  Reconnaissance  Wing,  Maintenance  Training  Management 

AF  Reserves 

924th  Consolidated  Aircraft  Maintenance  Squadron,  Training  Management,  Aircraft 
Propulsion  and  Aircraft  Maintenance  Sections 

AF  Reserves 

924th  Tactical  Fighter  Group  Consolidated  Base  Personnel  Office  Base  On-the-Job 
Training  Management  &  Quality  Force  Sections 

AF  Reserves 

924th  Weapons  System  Security  Flight,  All  Sections 

Air  National 
Guard 

147th  Consolidated  Aircraft  Maintenance  Squadron,  Training  Management,  Unit 
Administration,  Aircraft  Propulsion  and  Aircraft  Maintenance  Sections 

Air  National 
Guard 

147th  Fighter  Interceptor  Group,  Base  On-the-Job  Training  Management  &  Quality 
Force  Sections 

Air  National 
Guard 

147th  Security  Police  Right,  All  Sections 

*  Control  groups  identified  in  91st  and  45th  AMU 


The  implementation  schedule  was  as  follows: 

1  August  1988  Active  Duty  67th  Component  Repair  Squadron,  Jet  Engine  Inter¬ 
mediate  Maintenance  Work  center 

15  August  1988  All  remaining  Active  Duty  work  centers 

1  September  1988  All  Reserve  and  Air  National  Guard  work  centers. 

As  work  centers  were  implemented,  instruction  for  users  of  the  prototype  AOTS  was 
provided  by  AFHRL  and  contractor  personnel.  To  ensure  priority  was  given  to  the  primary 
users  of  the  prototype  AOTS,  supervisors  and  training  managers  were  trained  first,  fol¬ 
lowed  by  trainers,  evaluators,  trainees,  test  control  officers  (TCO)  and  quality  control 
administrators.  A  ratio  of  one  instructor  to  one  user  was  applied  during  training  to  ensure 
users  learned  to  operate  the  system  at  their  own  best  pace.  The  initial  intent  was  to 
instruct  supervisors  and  training  managers  on  operating  the  system,  and  in  turn,  have  the 
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supervisors  and  training  managers  provide  instruction  to  other  work  center  personnel. 
However,  other  work  center  personnel  were  not  receiving  adequate  and  timely  instruction. 
Therefore,  responsibility  for  all  remaining  instruction  was  assumed  by  AFHRL  and  con¬ 
tractor  personnel.  Copies  of  the  Operational  Guide  for  the  Prototype  AOTS  were  made 
available  to  users  to  enhance  their  understanding  of  system  capabilities  and  how  the  system 
was  intended  to  be  used.  In  addition,  User  Handbooks  for  the  Prototype  AOTS  were 
developed,  published  and  distributed  to  explain  how  training  functions  were  performed 
using  the  prototype  AOTS. 


Five  hundred  eighty-five  individuals  were  identified  to  participate  in  the  AOTS 
experiment.  The  participants  included  training/evaluation  developers,  supervisors,  train¬ 
ers,  evaluators,  trainees,  training  managers,  quality  control  monitors,  system  administrators, 
and  commanders  as  described  here: 

a.  Training  Developers  -  responsible  for  analyzing  and  documenting  performance 
requirements,  and  developing  behavioral  objectives,  tests  and  instructions; 

b.  Supervisors  -  responsible  for  ensuring  training  programs  were  properly 
planned  and  executed,  that  training  requirements  were  defined  for  each 
trainee  and  that  progress  and  training  history  records  were  maintained  for 
each  trainee; 

c.  Trainers  -  responsible  for  interacting  with  trainees  to  teach  knowledge,  skills 
and  procedures  associated  with  speciality  tasks; 

d.  Evaluators  -  responsible  for  interacting  with  trainees  to  either  administer 
knowledge  test  or  to  observe  and  rate  performance  on  specialty  tasks; 

e.  Trainees  -  recipient  of  job-site-training; 

f.  Training  Managers  -  responsible  for  overall  management  of  training  for  orga¬ 
nizations  and  assisting  supervisors  with  the  development  of  work  center  train¬ 
ing  programs; 

g.  Quality  Control  Administrators  -  responsible  for  scheduling  periodic  perform¬ 
ance  evaluations  to  determine  training  program  effectiveness  and  advise  com¬ 
manders  of  results;  and 

h.  Commanders  -  responsible  for  ensuring  effective  training  programs  were 
established  within  all  applicable  work  centers. 


To  ensure  the  AOTS  participants’  views  on  training  were  representative  of  a  larger 
Air  Force  sample,  a  training  views  survey  which  had  been  previously  administered  in  1987 
at  22  AF  bases  across  four  Major  Commands  (MAJCOMs)  was  administered  to  the  major¬ 
ity  of  AOTS  participants  during  June  and  July  1988.  The  number  of  individuals  included  in 
the  1987  sample  and  the  AOTS  sample  are  reported  in  Table  2.2  by  training  level. 
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TABLE  2.2 


NUMBER  OF  SUBJECTS  RESPONDING  TO  TRAINING  VIEWS  SURVEY 


1987  Sample  N 

AOTS  Sample  N 

Commander 

35 

11 

Training  Manager 

154 

12 

Supervisor,  Evaluator,  Trainer 

639 

232 

Trainee 

1200 

419 

The  responses  from  the  AOTS  sample  were  similar  to  the  larger  group  responding  in 
1987.  Thus,  the  AOTS  sample  was  considered  representative  of  the  general  AF  OJT 
population.  Both  groups  described  AF  training  as  follows: 

a.  involves  one-on-one  training; 

b.  limited  training  materials  available; 

c.  viewed  as  labor  intensive; 

d.  lack  of  standardized  evaluation;  and 

e.  important  to  mission  requirements. 
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3  AOTS  EVALUATION  FRAMEWORK 


A  major  objective  of  the  prototype  AOTS  project  was  to  present  data  required  for  the 
Air  Force  to  determine  whether  the  system  provided  capabilities  to  overcome  deficiencies 
identified  in  the  conventional  OJT  system,  and  if  the  system  should  be  implemented  across 
the  operational  environment.  To  facilitate  the  presentation  of  required  data,  strategies 
were  developed  for  testing  and  evaluating  the  system  and  for  reporting  results.  These  strat¬ 
egies  were  documented  in  a  Master  Test  Plan  (MTP).  The  MTP  is  discussed  later  in  this 
section. 


Strategies  were  developed  for  both  formative  and  summative  evaluations.  Formative 
evaluation  occurred  during  development  of  the  system.  The  purpose  was  to  determine 
whether  specifications  had  been  met,  and  to  provide  feedback  to  designers  and  implement¬ 
ed  which  enabled  them  to  make  necessary  changes  and  adjustments.  Summative  evalu¬ 
ation  took  place  after  the  system  had  been  implemented  in  the  operational  environment. 
The  purpose  was  to  determine  if  the  system  functioned  correctly,  what  impacts  it  had  on 
training  programs,  and  how  it  was  perceived  by  users  and  observers.  AFHRL  identified 
four  critical  issues  to  be  focused  upon  during  the  evaluation  of  the  prototype  AOTS.  These 
critical  issues  were  1)  Compliance,  2)  Performance,  3)  Acceptance  and  4)  Suitability.  The 
development  of  evaluation  strategies  included  developing  objectives  for  the  critical  issues 
and  establishing  procedures  for  collecting,  analyzing  and  reporting  data. 

During  the  design  and  implementation  of  the  prototype  AOTS,  capabilities  were  pro¬ 
vided  for  generating  data  required  for  system  assessment.  However,  data  required  for 
assessing  the  conventional  OJT  system  were  not  available.  When  it  was  deemed  necessary 
to  compare  the  prototype  AOTS  to  the  conventional  OJT  system,  elements  for  data  rele¬ 
vant  to  the  conventional  system  had  to  be  defined  and  procedures  for  collecting  data  and 
reporting  results  had  to  be  developed.  Further,  when  evaluation  strategies  included 
surveying  and/or  interviewing  users  and  observers  of  the  prototype  AOTS,  appropriate 
instruments  and  procedures  for  administration  were  developed. 


A  classic,  widely  used,  three  dimensional  model  from  the  program  evaluation  litera¬ 
ture  was  tailored  to  provide  the  framework  for  the  evaluation  of  the  prototype  AOTS.  This 
was  the  Criteria  Acquisition  Model  (CAM)'  illustrated  in  Figure  3.1. 
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Figure  3.1.  AOTS  Tailored  Criteria  Acquisition  Model 

A  primary  consideration  in  selecting  this  model  was  that  it  supports  systematic,  com¬ 
prehensive  evaluation  of  a  large  scale  system,  such  as  the  prototype  AOTS.  Additionally, 
factors  to  be  considered  duringthe  evaluation  of  the  AOTS  could  be  easily  correlated  to 
the  dimensions  of  the  model.  The  two  stages  in  the  model  referred  to  Parts  1  and  2  of  the 
MTP.  The  audiences  referred  to  the  developers  and  managers  of  AOTS  data;  Air  Force 
managers  such  as  MAJCOM  observers,  commanders  and  training  managers;  and  end  users 
of  AOTS  products  such  as  supervisors,  trainees,  trainers,  and  evaluators.  The  critical  issues 
referred  to  the  four  critical  issues  identified  to  assess  the  success  of  AOTS.  Definitions  of 
the  four  critical  evaluation  issues  were  as  follows: 

a.  Compliance  Did  the  prototype  AOTS  meet  the  design  and  functional 

requirements  oi  the  system,  subsystem,  and  component  speci¬ 
fications? 

b.  Performance  Did  AOTS  meet  the  system  performance  standards?  How 

did  performance  of  the  prototype  AOTS  compare  with  the 
performance  of  conventional  OJT  systems? 

c.  Acceptance  Was  the  AOTS  accepted  by  the  various  system  users  as  user 

friendly,  i.e.,  easy  to  use? 

d.  Suitability  Did  the  prototype  AOTS  overcome  currently  defined  defi¬ 

ciencies  in  the  Air  Force’s  OJT  system?  Could  AOTS  be 
used  for  OJT  throughout  the  Air  Force? 


1  Wright,  W.J.,  &  Hess,  R.J.  (1974),  A  criteria  acquisition  model  for  education  evaluation.  !n  G.  D.  Borich  (Ed.),  Evaluation 
Education  Programs  and  Products.  Englewood  Cliffs,  NJ:  Education  Technology  Publications. 
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A  multifaceted  evaluation  scheme  was  developed  to  evaluate  the  AOTS.  A  two  part 
MTP,  monitored  by  a  quarterly  working  group,  was  designed  to  evaluate  the  AOTS  during 
system  development  (i.e.,  Phase  2)  and  during  AOTS  implementation  in  the  operational 
work  centers  (i.e.,  Phase  3).  Procedures  outlined  in  Part  1  and  Part  2  of  the  MTP  are  sum¬ 
marized  below. 

3.1  Master  Test  Plan  Part  1  -  Functional  Testing 

Part  1  of  the  MTP  outlined  procedures  for  verifying  the  attainment  of  design  objec¬ 
tives  for  functions,  components,  subsystems,  interfaces,  and  technical  performance  specifi¬ 
cations  before  the  AOTS  was  implemented  into  operational  work  centers.  Test  Plans  were 
developed  for  alpha,  beta,  and  readiness  testing. 


Alpha  testing  was  accomplished  by  contractor  personnel  other  than  software  engi¬ 
neers  who  had  developed  the  software.  The  purpose  for  alpha  testing  was  to  determine 
whether  the  software  functioned  properly  on  the  hardware  selected  to  host  the  prototype 
AOTS,  and  whether  the  software  and  hardware  together  performed  the  functions  required 
by  the  system,  subsystem  and  component  specifications.  Alpha  testing  encompassed  unit, 
integration  and  system  level  testing.  During  alpha  testing,  a  go/no  go  approach  was 
employed.  This  meant  that  either  a  computer  program  performed  correctly  or  it  was  not 
released  for  operational  use.  Repairs  were  made  until  such  time  as  the  software  functioned 
properly. 


Beta  testing  was  accomplished  by  AFHRL  personnel,  after  each  software  release. 

The  purpose  for  beta  testing  was  to  determine  whether  software  programs  functioned  pro¬ 
perly  when  operated  by  personnel  familiar  with  the  environment  in  which  the  system  would 
reside,  and  whether  products  generated  for  off-line  use  were  correctly  formatted  and 
contained  appropriate  data.  Testing  was  accomplished  through  operating  on-line  capabili¬ 
ties  for  specified  periods  of  time.  Discrepancies  were  noted  and  revisions  to  software  were 
prioritized  and  made  in  order  of  importance  to  the  users. 


Readiness  testing  was  accomplished  after  beta  testing  by  Air  Force  and  contractor 
personnel  who  had  not  directly  participated  in  the  actual  design  and  development  of  the 
prototype  AOTS.  The  purpose  for  readiness  testing  was  to  ensure  that  all  software  and 
hardware  functioned  properly  and  was  suitable  for  implementation  in  the  operational  work 
centers.  The  results  of  the  readiness  test  confirmed  that  the  prototype  AOTS  was  ready  to 
be  phased  into  the  previously  identified  work  centers. 


3.2  Master  Test  Plan  Part  2  -  System  Level  Test  and  Evaluation 


Part  2  of  the  MTP  outlined  procedures  for  the  system  level  test  and  evaluation 
(SLT&E)  of  the  prototype  AOTS  in  the  operational  work  centers.  Procedures  were  devel¬ 
oped  to  collect  baseline  and  control  data  tor  comparison  with  the  AOTS  data  as  illustrated 
in  Figure  3.2. 
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TREATMENT  GROUP 

CONTROL  GROUP 

Figure  3.2.  Prototype  AOTS  Experimental  Design 


AOTS  DATA 


BASELINE  DATA 


Part  2  of  the  MTP  called  for  data  to  be  collected: 

a.  on  and  from  individuals  assigned  to  or  associated  with  the  treatment  group 
work  centers; 

b.  on  individuals  assigned  to  control  groups; 

c.  from  base-level  reports; 

d.  from  developers  of  training  and  evaluation  materials;  and 

e.  from  observers  of  the  prototype  AOTS. 


As  the  project  progressed,  it  became  apparent  that  revisions  to  the  MTP  were 
required  to  provide  a  more  complete  framework  for  conducting  the  SLT&E.  The  initial 
MTP  lacked  the  dimensional  perspective  and  flexibility  required  for  conducting  research  in 
a  "live"  environment.  Originally,  the  MTP  was  oriented  toward  outcome  data  with  minimal 
evaluation  of  who  used  the  system,  when  and  how  the  system  was  used,  and  what  back¬ 
ground  factors  influenced  the  utilization  of  the  system.  The  initial  focus  of  the  MTP  was  to 
evaluate  the  prototype  AOTS  functions  and  not  the  reciprocal  interactions  between  the 
AOTS  and  the  environment. 


There  were  two  basic  assumptions  during  the  MTP  development:  users  had  a  basic 
understanding  of  Air  Force  OJT  policies  and  procedures  as  prescribed  by  Air  Force  Regu¬ 
lation  50-23  and,  once  AOTS  was  deployed,  it  would  be  used  on  a  regular  basis  to  conduct 
OJT.  Neither  assumption  was  realized  to  the  level  expected.  Users  conducted  OJT  in  an 
unstructured  manner  with  limited  awareness  of  their  OJT  responsibilities,  and  the  system 
was  underutilized.  Because  of  these  violated  assumptions  and  the  need  to  assess  the  inter¬ 
action  between  the  AOTS  and  users,  the  MTP  evolved  into  a  more  responsive  and  multi¬ 
faceted  program  evaluation  approach. 


A  more  flexible  MTP  assessment  approach  was  developed  which  included  the  collec¬ 
tion  of  process  and  contextual  data  (see  the  MTP  for  specific  descriptions).  This  flexibility 
resulted  in  the  use  of  subsamples  representing  varying  levels  of  usage  of  the  AOTS.  For 
example,  a  subsample  of  higher  frequency  users  was  identified  and  interviewed  to  obtain 
more  valid  feedback  on  the  prototype.  Process  data  provided  information  on  who,  when 
and  how  the  prototype  system  was  being  implemented.  It  consisted  of  standardized  obser¬ 
vations  conducted  on  a  regular  basis  at  the  work  center  level,  documentation  of  informa¬ 
tion/assistance  calls  (help  hotline),  and  computer  records  of  system  use  by  individual  user 
identification  number.  Contextual  data  consisted  of  a  log  of  various  field  activities  during 
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the  deployment  period.  These  supplemental  measures  expanded  the  MTP  to  include  infor¬ 
mation  on  the  reciprocal  influences  of  the  operational  environment.  The  final  MTP  pro¬ 
vided  a  more  comprehensive,  flexible  and  multifaceted  assessment  of  automated 
technology  in  the  Air  Force  operational  setting. 
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4  SLT&E  DATA  COLLECTION  METHODOLOG'' 


The  methods  used  to  collect  the  AOTS  assessment  data  included:  1)  administering 
checklists,  surveys,  and  interviews;  2)  conducting  system  performance  tests;  3)  extracting 
data  from  maintenance  reports,  training  records,  quality  assurance  reports,  crime  statistics 
and  utilization  reports;  (4)  reporting  the  percentage  of  time  spent  by  commanders,  supervi¬ 
sors,  training  managers  and  trainers/evaluators  on  training  related  tasks;  (5)  capturing  pro¬ 
cess  data;  and  (6)  conducting  an  independent  study  to  compare  knowledge  and 
performance  scores  obtained  by  subjects  in  treatment  and  control  group  maintenance  work 
centers.  Figure  4.1  provides  a  visual  overview  of  the  AOTS  evaluation  data  sources. 


Figure  4.1.  AOTS  Test  and  Evaluation  Data  Sources 


4.1  Compliance  Data  Collection  Instruments 

To  determine  whether  the  AOTS  met  the  design  and  functional  requirements  of  the 
system  specifications,  computer  response  times  were  observed,  an  interface  checklist  was 
reviewed  and  hardware  re  iability,  maintainability,  and  availability  were  computed.  Each 
methodology  is  explained  here. 
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a.  COMPUTER  RESPONSE  TIME.  Computer  terminal  response  times  and 
search  response  times  were  observed  under  normal  AOTS  operating  condi¬ 
tions  three  months  after  the  start  of  the  AOTS  SLT&E  to  ensure  the  specified 
response  times  were  met.  Terminal  response  time  was  the  amount  of  time 
required  from  the  moment  a  keyboard  key  was  pressed  until  the  desired  dis¬ 
play  appeared  on  the  screen.  Search  time  was  the  amount  of  time  required 
from  the  moment  a  search  request  was  entered  until  the  information  was 
displayed  on  the  screen.  Five  trials  were  executed  for  each  response  time  pro¬ 
cedure.  The  results  of  the  five  trials  and  an  average  were  recorded  on  a  Per¬ 
formance  Response  Time  Form. 

b.  EXTERNAL  INTERFACE  CHECKLIST.  During  Phase  2,  interfaces 
between  AOTS  and  14  external  entities  (e.g.,  the  United  States  Air  Force 
Occupational  Measurement  Center,  the  Publication  Management  System,  and 
the  Aar  Force  Military  Personnel  Center)  were  identified  as  desirable.  A  list 
of  the  external  entities  was  displayed  in  a  matrix.  The  matrix  was  used  as  a 
checklist  to  report  whether  one  or  more  of  the  following  interface  types  had 
been  established  between  AOTS  and  each  respective  external  entity: 

1)  Manual  -  An  interface  totally  off  line 

2)  Automated  -  An  interface  performed  with  the  system  using  a  keyboard, 
touch  pad,  and/or  an  optical  mark  reader 

3)  Automatic  -  An  interface  performed  entirely  by  the  computer  without 
additional  data  input  by  the  user. 

The  checklist  was  reviewed  by  three  contractor  and  three  Air  Force  repre¬ 
sentatives  eight  months  after  the  start  of  the  AOTS  SLT&E.  The  reviewers 
collectively  recorded  "YES"  if  the  interface  was  believed  to  have  been 
established,  and  "NO"  if  the  interface  was  believed  to  have  remained  unestab¬ 
lished. 

c.  EQUIPMENT  RELIABILITY,  MAINTAINABILITY  AND  AVAILABIL¬ 
ITY.  Goals  were  established  and  recorded  on  a  table  before  the  start  of 
AOTS  SLT&E.  Representative  operational  data  were  gathered  during  normal 
operational  use  of  the  equipment  between  1  October  1986  and  1  February 
1989.  The  actual  reliability,  maintainability,  and  availability  values  were  cal¬ 
culated  and  compared  to  the  goals  set  before  the  start  of  SLT&E.  The  typical 
work  center  work  station  included  a  Zenith{TM)-248  Personal  Computer, 
Scantronw  model  5200  optical  mark  reader,  AlpSftv,  P2000G  printer;  and  in 
some  work  centers,  a  Sony  LDP2000/4  interactive  disc  player,  an  Electrohome 
1300  color  monitor  with  touch  screen,  and  sound.  Dedicated  high-speed  digi¬ 
tal  communication  lines  were  leased  for  transmission  between  the  Digital™ 
VAX  8650  host  computer  at  Brooks  Air  Force  Base  and  terminals  at 
Bergstrom  AFB  and  Ellington  Air  National  Guard  Base.  Infotron  Systems™ 
992NP  multiplexers  were  used  to  control  and  distribute  communications  at  the 
bases. 

1)  HARDWARE  RELIABILITY.  Reliability  was  expressed  as  the  Mean 
Time  Between  Failures  (MTBF),  in  hours,  that  a  component  or  system 
operated  without  a  failure  during  normal  operational  use  of  the  equip¬ 
ment.  Data  to  determine  the  reliability  of  work  center  equipment  (i.e., 
personal  computer,  monitor,  printer  and  optical  mark  reader)  were 
extracted  from  the  prototvpe  AOTS  Maintenance  Action  Report  Log 
and  the  prototype  AOTS  Procedure:  Maintenance  Log. 
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2)  HARDWARE  MAINTAINABILITY.  Maintainability  was  expressed  as 
the  Mean  Time  To  Repair  (MTTR)  a  component  or  system  to  full 
operational  condition.  The  data  were  extracted  quarterly  from  the  pro¬ 
totype  AOTS  Maintenance  Action  Reports,  the  Communication  Prob¬ 
lem  Log  and  the  VAX  Down  Time  Log. 

3)  SYSTEM  AVAILABILITY.  Availability  was  the  percentage  of  time  the 
prototype  AOTS  equipment  was  available  for  use  as  opposed  to  down 
time.  The  percentage  was  obtained  by  dividing  the  total  time  the  system 
was  up  and  operational  by  the  total  time  the  system  was  required  to  be 
up  and  operational. 


4.2  Performance  Data  Collection  Instruments 

Unit  assessment  data  and  individual  airman  training  program  data  were  used  to  com¬ 
pare  performance  under  AOTS  with  performance  under  conventional  OJT.  The  purpose 
was  to  determine  whether  the  prototype  AOTS  had  an  impact  on  unit  performance  and 
trainee  progress  toward  duty  position  qualification. 

4.2. 1  Unit  Assessment  Instruments  Designed  To  Address  Performance. 

Maintenance  quality  assurance  results  and  the  number  of  repeat  maintenance  actions 
were  extracted  from  two  base  level  maintenance  reports.  Security  police  quality  control 
average  scores  and  crime  statistics  were  extracted  from  two  security  police  base-level  quar¬ 
terly  reports.  The  frequency  of  extraction  and  the  instruments  used  to  collect  the  respec¬ 
tive  data  are  identified  here.  Data  to  determine  personnel  work  center  performance  were 
unavailable. 


a.  MAINTENANCE  QUALITY  ASSURANCE  DATA.  Aircraft  Maintenance 
Unit  (AMU),  Propulsion,  and  Maintenance  Branch  Quality  Assurance  (QA) 
Data  were  collected  quarterly  before  and  during  the  AOTS  SLT&E.  Tlie 
Quality  Assurance  Program  Monthly  Summary  Data  Recording  Forms  were 
used  to  report  technical  inspection  and  personnel  evaluation  results  extracted 
from  the  Air  Force  active  duty  67  TRW  Quality  Assurance  Quarterly  Sum¬ 
mary,  the  Air  Force  Reserve  924  TFG  Quality  Assurance  Quarterly  Summary, 
and  Air  National  Guard  AF  Form  2419.  Reserve  and  Air  National  Guard 
data  were  extracted  for  the  Aircraft  Maintenance  and  Propulsion  branches. 
Active  duty  data  were  extracted  for  the  AMU,  Propulsion,  and  Inspection 
branches. 

b.  REPEAT  MAINTENANCE  ACTIONS  DATA.  Aircraft  Maintenance  Data 
extracted  monthly  before  and  during  SLT&E  from  Maintenance  Data  Collec¬ 
tion  (MDC)  and  Maintenance  Information  Logically  Analyzed  and  Presented 
(MILAP)  480  reports  generated  by  the  67th  TRW  DCM/MASA  office  were 
reported  on  the  MDC/MILAP  Data  Collection  Form.  The  data  extracted 
from  the  MDC  documents  reported  the  number  of  repeat  maintenance  actions 
performed  at  five  work  centers.  The  data  extracted  from  the  MILAP  480 
reports  represented  the  number  of  repeat  actions  at  the  aircraft  level  as 
reported  by  pilots  flying  the  respective  aircraft. 
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c.  SECURITY  POLICE  QUALITY  CONTROL  DATA.  The  Security  Police 
Quality  Control  Trend  Data  Recording  Form  was  used  to  report  average 
scores  extracted  from  the  Security  Police  Quality  Control  Analysis  report.  The 
data  were  collected  quarterly  before  and  during  the  prototype  AOTS  SLT&E. 

d.  BASE  CRIME  DATA.  Base  Crime  Statistics  were  collected  and  reported 
quarterly  before  and  during  the  SLT&E.  Data  were  extracted  from  the  Base 
Crime  Analysis  Report  ana  recorded  on  the  Security  Police  Crime  Analysis 
Data  Collection  form. 


4.2.2  Airman  Training  Progress  Data  To  Measure  Performance 

Four  methods  were  used  to  collect  data  relevant  to  individual  airman  training  pro¬ 
grams.  Task  training  record  data  were  extracted  from  AF  Forms  623  during  the  baseline, 
interval  and  from  AOTS  Airman  Training  Records  during  SLT&E.  Time  available  and 
time  spent  on  training  related  tasks  were  recorded  on  Weekly  Inventory  of  Time  Spent 
(WITS)  forms  before  and  during  SLT&E.  Knowledge  and  performance  test  results  were 
captured  towards  the  end  of  the  prototype  AOTS  SLT&E.  Survey  and  interview  responses 
were  captured  during  and  at  the  end  of  SLT&E.  All  four  methods  are  described  below. 

a.  ON-THE-JOB  TRAINING  DATA.  OJT  Data  Recording  Forms  were  used  to 
report  individual  trainee  baseline  data  extracted  from  AF  Forms  623.  Data 
obtained  from  the  forms  were  recorded  at  the  beginning,  midway,  and  at  the 
end  of  the  baseline  data  collection  effort.  The  extracted  data  included: 

1)  the  number  of  tasks  circled  (i.e.  the  number  of  tasks  required  for  an  indi¬ 
vidual  to  be  duty  position  qualified) 

2)  the  number  of  tasks  opened  each  month  during  the  reporting  period  (i.e. 
the  number  of  tasks  being  trained) 

3)  the  number  of  tasks  closed/completed  each  month  during  the  reporting 
period  (i.e.  the  number  of  tasks  certified) 

4)  the  date  the  trainee  was  position  qualified  (i.e.  the  date  the  last  task  was 
completed  indicating  that  all  circled  tasks  were  certified). 

Individual  Airman  Training  data  were  automatically  collected  by  the  system 
during  the  AOTS  SLT&E.  The  data  included  the  number  of  tasks  required 
for  an  individual  to  be  duty  position  qualified,  as  well  as  the  number  and  per¬ 
centage  of  the  duty  position  required  tasks  that  the  airman  was  certified  to 
perform.  The  data  were  captured  monthly  from  October  1988  through  June 
1989. 

b.  WEEKLY  INVENTORY  OF  TIME  SPENT  (WITS).  Treatment  and  control 
group  subjects  recorded  the  number  of  hours  available  and  the  number  of 
hours  spent  performing  training  related  activities  before  and  during  AOTS 
SLT&E.  The  data  were  collected  for  a  seven-day  period  every  other  week 
from  the  Active  duty  personnel.  The  Reserve  and  Air  National  Guard  person¬ 
nel  documented  their  hours  during  Unit  Training  Assembly  (UTA)  weekends 
and  while  on  annual  tour. 
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c.  TEST  AND  EVALUATION  SURVEY  AND  INTERVIEW  RESPONSES. 
Test  and  Evaluation  survey  data  relevant  to  AOTS  performance  were  col¬ 
lected  from  training  managers,  supervisors,  trainers,  and  evaluators  via  ques¬ 
tionnaires  and  interviews  midway  and  at  the  end  of  SLT&E.  The  instruments 
were  intended  to  provide  information  as  to  whether  the  AOTS  was  better, 
worse,  or  no  different  than  conventional  on-the-job  training. 

d.  PERFORMANCE  MEASUREMENT  STUDY.  Maintenance  knowledge  and 
performance  tests  were  administered  to  tactical  aircraft  maintenance  and 
aerospace  propulsion  airmen  who  were  previously  certified  on  tasks  performed 
by  members  of  their  respective  specialty.  The  purpose  of  the  evaluations  was 
to  determine  whether  maintenance  personnel,  who  viewed  Computer  Based 
Training  (CBT)  modules,  demonstrated  having  more  task  related  knowledge 
and  performed  tasks  better  than  personnel  who  did  not  view  the  CBT  mod¬ 
ules. 


4.3  Acceptance  and  Suitability  Data 

Opinion  surveys  and  interviews  were  administered  to  address  the  perceived  accep¬ 
tance  and  suitability  of  the  prototype  AOTS.  The  surveys  were  intended  to  provide  infor¬ 
mation  as  to  whether  AOTS  participants  believed  the  AOTS  was  easy  to  use  and  suitable 
for  use  in  the  operational  environment 


The  survey  response  scale  ranged  from  low  (a=  1)  to  high  (g=7).  For  the  purpose  of 
conveying  SLT&E  findings  via  bar  charts,  the  scale  was  clustered  into  three  major  catego¬ 
ries:  1-3,  unfavorable;  4,  neutral;  5-7,  favorable.  Developers  responded  to  their  survey  one 
time,  three  months  after  the  start  of  SLT&E.  Surveys  were  administered  to  supervisors, 
training  managers,  trainers,  evaluators,  and  trainees  midway  through  and  at  the  end  of 
AOTS  SLT&E.  Commanders  responded  to  an  AOTS  opinion  survey  one  time  approxi¬ 
mately  eight  months  into  the  evaluation.  Third  Party  MAJCOM  Observers  were  adminis¬ 
tered  opinion  surveys  one  time  between  January  and  April  1989  after  a  two  day  orientation 
of  the  AOTS. 


Table  4.1  indicates  the  category  of  individuals  responding  to  the  test  and  evaluation 
surveys,  the  number  of  items  included  on  each  respective  survey,  the  frequency  of  adminis¬ 
tration,  and  the  number  of  individuals  returning  usable  survey  response  sheets  at  each 
administration. 
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TABLE  4.1 

TEST  AND  EVALUATION  SURVEY  CHARACTERISTICS 


RESPONDENT 

CATEGORY 

NUMBER 

OF  ITEMS 

ADMIN 

FREQUENCY 

NUMBER  OF 
RESPO 

RESPONDING 

NDENTS 

TIME  1 

TIME  2 

SUPERVISOR 

116 

2 

114 

59 

TRAINING  MANAGER 

87 

2 

16 

16 

TRAINEE 

39 

2 

82 

46 

COMMANDER 

54 

1 

11 

N/A 

DEVELOPER 

20 

1 

8 

N/A 

MAJCOM  OBSERVER 

37 

1 

82 

N/A 

4.4  Process  Data 


Process  data  were  collected  to  help  the  researchers  understand  the  results  of  the  cap¬ 
tured  SLT&E  data.  Process  data  were  obtained  by  identifying  and  monitoring  AOTS-user- 
environment  information  concerning  the  utilization  of  the  system  in  the  operational  setting 
and  by  documenting  the  occurrence  of  base  activities  during  the  SLT&E  interval.  Process 
data  consisted  of  the  following: 

a.  Context  Information-  field  activities  occurring  during  the  SLT&E  period  of  1 
August  1988  to  31  July  1989  (e.g.,  base  exercises,  ORI,  major  base  activities), 

b.  Observations-  380  work  center  observations  documented  on  a  standardized  form 
by  Instructional  Systems  Team  (1ST)  personnel, 

c.  Hotline  calls-  225  telephone  calls  by  work  center  personnel,  asking  for  assistance 
in  using  the  prototype  AOTS,  and 

d.  Log-on  records-  user  requested  computer  generated  records  of  AOTS  use  sorted 
by  individual  access  code  number. 


Figure  4.2  summarizes  the  MTP  development  and  data  collection  schedule. 
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PHASE  I  w  PHASE  11  ^  PHASE  111  v  w 


Figure  42.  AOTS  MTP  Development  and  Data  Collection  Schedule 

As  noted  on  Figure  4.2,  the  Master  Test  Plan  covered  all  phases  of  the  AOTS  project. 
Baseline  and  training  view  data  were  collected  to  provide  a  benchmark  against  which  the 
impact  of  the  AOTS  on  the  existing  AJF  OJT  could  be  measured.  Phase  III  Process  Data 
provided  information  on  the  implementation  of  the  system.  These  data  as  well  as  outcome 
measures  were  used  to  address  the  critical  issues  of  interest  to  this  study. 
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5  RESULTS 


This  section  provides  discussions  of  the  results  from  analyzing  data  collected  before 
and  during  the  SLT&E  period.  Discussions  are  presented  by  critical  issue. 

5.1  Compliance  Results 

The  AOTS  met  the  system  specification  while  operating  in  an  operational  environ¬ 
ment:  1)  performance  response  times  were  met  and  exceeded;  2)  appropriate  external 
interfaces  were  established;  and  3)  all  equipment  goals  were  met  and  exceeded  as  reported 
here. 

5.1.1  Computer  Response  Times  / 

The  average  terminal  response  and  Master  Task  List  (MTL)  search  times  observed 
under  normal  AOTS  operating  conditions  were  2  and  56  seconds  respectively.  As  shown  in 
Table  5.1,  both  average  times  were  within  the  specified  acceptable  levels. 


Table  5.1 

AOTS  AVERAGE  TERMINAL  AND  MTL  SEARCH  TIMES  IN  SECONDS 


OPERATION 

ACCEPTABLE  AVERAGE 
TIME 

ACTUAL  AVERAGE 

TIME 

TERMINAL  RESPONSE 

02.50  Seconds 

02.00  Seconds 

MTL  SEARCH 

60.00  Seconds 

56.00  Seconds 

5. 1 .2  External  Interfaces 

Interfaces  were  established  between  the  AOTS  and  eight  external  entities.  Table  5.2 
reports  the  type  (i.e.,  manual,  automated,  or  automatic)  and  descriptions  of  the  established 
interfaces,  and  indicates  the  reasons  for  deleting  the  interface  requirements  between  the 
AOTS  and  six  external  entities. 
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Table  5.2 

EXTERNAL  INTERFACE  CHECKUST 


EXTERNAL  INTERFACE 
ENTITY 

INTERFACE 

YES/NO 

METHOD  OF 
INTERFACE 

COMMENTS 

UNITED  STATES  AJR 

FORCE  OCCUPATIONAL 
MEASUREMENT  CENTER 

YES 

''MANUAL/ 

2AUTOMATIC 

SURVEY  DATA  ARE  ELECTRONICALLY  PASSED 

FROM  SPERRY  TO  VAX;  AOTS  AUTOMATICALLY 
PULLS  DATE  SUCH  AS  TASK  STATEMENT  AND  TASK 
FACTORS  ANALYSIS;  AOTS  MANUALLY  PROVIDES 
USAFOMC  WITH  INFORMATION  TO  VAUDATE  TASK 
LIST 

PUBLICATION  MANAGEMENT 
SYSTEM 

YES 

„  MANUAL/ 
3AUTOMATED 

AOTS  REPRESENTATIVES  MANUALLY  OBTAIN  PUB¬ 
LICATIONS;  AOTS  ALLOWS  FOR  AUTOMATED 
LOADING  OF  ID  FOR  PUBS  GOVERNING  AOTS 

TASKS  AND  REVISED  PUBS  THAT  MIGHT  EFFECT 
TASK  TRAINING  • 

AUTOMATED  TECHNICAL 
ORDER  SYSTEM  (ATOS) 

NO 

ATOS  CURRENTLY  NOT  IMPLEMENTED  AT  THIS 

AFB,  INTERFACE  IS  NOT  POSSIBLE 

PERSONNEL  DATA  SYSTEM 
(PDS)  PERSONNEL  CONCEPT 

III  (PC3) 

YES 

MANUAL/ 

AUTOMATIC 

(MAGNETIC 

TAPE) 

AUTOMATED 

AOTS  MANUALLY  RECEIVES  MAGNETIC  TAPES 

FROM  PDS;  AOTS  SOFTWARE  READS  MAGNETIC 
TAPE  AND  AUTOMATICALLY  PULLS  DATA  SUCH  AS 
DUTY  POSITION  AND  CONTROL  AFSC  INTO  AOTS; 
DATA  ALLOWS  FOR  AUTOMATED  LOADING  OF 
MANPOWER  DATA  SUCH  AS  POSITION  NUMBERS 

SECURITY  POLICE  AUTOMA¬ 
TED  SYSTEM  (SPAS) 

NO 

AOTS  ALLOWS  FOR  AUTOMATED  LOADING  OF 

TASK  AND  OTHER  TRAINING  REQUIREMENTS  DATA 
ONCE  THE  DATA  ARE  INCLUDED  IN  SPAS  AND  THE 
DATA  ARE  MANUALLY  DELIVERED  TO  AOTS  (TASK 
AND  OTHER  TRAINING  REQUIREMENTS  DATA  ARE 
NOT  YET  INCLUDED  IN  SPAS) 

CORE  AUTOMATED  MAINTE¬ 
NANCE  SYSTEM  (CAM) 

YES 

MANUAL/ 

AUTOMATED 

AOTS  MANUALLY  RECEIVES  AND  ALLOWS  FOR 
AUTOMATED  LOADING  OF  TASK  QUALIFICA¬ 
TION/TRAINING  REQUIREMENTS  IDENTIFIED  IN 
CAMS;  AOTS  MANUALLY  PROVIDES  CAMS  WITH 
UPDATED  DATA 

OPERATIONAL  UNITS 

YES 

AUTOMATED/ 

AUTOMATIC 

AOTS  ALLOWS  FOR  AUTOMATED  TRAINING  MAN¬ 
AGEMENT  AND  DATA  UPDATES;  AOTS  AUTOMATI¬ 
CALLY  GENERATES  EVENT  NOTICES.  TRAINING 

AND  EVALUATION  RESULTS,  AND  REPORTS 

1  MANUAL 

2  AUTOMATIC 

3  AUTOMATED 


. INTERFACE  PROCESS  KEY . 

PROCESS  PERFORMED  EITHER  PARTIALLY  OR  TOTALLY  OFF  LINE 

PROCESS  PERFORMED  ENTIRELY  BY  THE  COMPUTER  ONCE  THE  COMMANDS 
HAVE  BEEN  ENTERED 

PROCESS  PERFORMED  BY  USER  INTERACTION  WITH  THE  SYSTEM  USING  A 
KEYBOARD,  TOUCH  PAD,  OR  RUNNING  A  FORM  THROUGH  AN  OPTICAL  MARK  READER 
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Table  5.2  Cont. 

EXTERNAL  INTERFACE  CHECKUST 


EXTERNAL  INTERFACE 
ENTITY 

INTERFACE 

YES/NO 

METHOD  OF 
INTERFACE 

COMMENTS 

AIR  FORCE  MILITARY  PERSON¬ 
NEL  CENTER  (AFMPC) 

YES 

MANUAL 

AFMPC  PROVIDES  POLICY  CHANGES  AND  AFS 
CONVERSION  DATA  SO  USERS  CAN  PREPARE  FOR 
CHANGES  BEFORE  THE  CHANGES  OCCUR 

EXTENSION  COURSE  INSTI¬ 
TUTE  (ECI) 

YES 

MANUAL/ 

AUTOMATED/ 

AUTOMATIC 

ECI  MANUALLY  TELLS  AOTS  WHEN  COURSES  ARE 
AVAILABLE/CHANGING;  AOTS  ALLOWS  FOR 
AUTOMATED  LOADING  OF  COURSES;  AOTS  AUTO¬ 
MATICALLY  GENERATES  NOTICES  OF  ELIGIBILITY 

COMPUTER  ASSISTED  TRAIN¬ 
ING  SYSTEM  (CATS) 

NO 

AOTS  ALLOWS  FOR  AUTOMATED  LOADING  OF  CAI 
LESSONS  IDs  (THE  LESSONS  ARE  NOT  AVAILABLE 

IN  AOTS) 

OFF-LINE  TRAINING  ACTIVITIES 

YES 

MANUAL/ 

AUTOMATED 

AOTS  MANUALLY  RECEIVES  COURSE  DATA;  AOTS 
ALLOWS  FOR  AUTOMATED  LOADING  OF  COURSE 
DATA  AND  GENERATION  OF  SCHEDULES  FOR 
ANCILLARY,  CONTINGENCY  AND  FORMAL  TRAIN¬ 
ING  COURSES 

TRAINING  DECISIONS  SYSTEM 
(TDS) 

NO 

TDS  CURRENTLY  NOT  OPERATIONAL;  HOWEVER, 
AOTS  PROVIDES  CAPABILITY  FOR  AUTOMATED 
LOADING  OF  TASK  TRAINING  MODULE  INFORMA¬ 
TION  IN  THE  MTL  TASK  RECORDS 

ADVANCED  TRAINING  SYSTEM 
(ATS) 

NO 

ATS  CURRENTLY  NOT  DEVELOPED 

BASE  SUPPLY 

NO 

REQUIREMENT  DELETED 

5.1.3  Hardware  Reliability 

All  reliability  goals  were  met.  As  shown  in  Table  5.3,  the  actual  Mean  Time  Between 
Failures  (MTBF)  was  more  than  double  the  established  goal  of  250  hours.  The  actual 
Mean  Time  Between  Downing  Events  (MTBDE)  was  two  thirds  higher  than  the  specified 
goal  of  400  hours,  and  Mission  Reliability  exceeded  the  established  goal  by  more  than  two 
percent.  The  reliability  coefficient  calculated  to  predict  the  probability  that  the  four  work 
center  components  (computer,  monitor,  printer,  and  optical  mark  reader)  would  operate 
successfully  in  the  operational  environment  during  an  eight  hour  shift  was  .8915  at  a  .9  con¬ 
fidence  factor. 
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TABLE  5.3 

SYSTEM  RELIABILITY  GOALS  AND  ACTUAL  VALUES 


PARAMETER 

SPECIFIED  GOAL 

ACTUAL 

Mean  Time  Between  Failures  (MTBF) 

250  Hours 

562  Hours 

Mean  Time  Between  Downing  Events 
(MTBDE) 

400  Hours 

673  Hours 

Mission  Reliability 

96.5% 

98.93% 

5. 1 .4  Hardware  Maintainability 

Table  5.4  displays  the  maintainability  goals  established  before  the  start  of  SLT&E 
and  the  actual  maintainability  values  obtained  during  the  deployment  of  the  prototype 
AOTS.  All  maintainability  goals  except  Mean  Administrative  Delay  Time  (A^t)  and  Mean 
Maintenance  Down  Time  (M^t) were  met-  Mean  Maintenance  Down  Time  incorporated 
Mean  Administrative  Delay  Time.  Thus,  the  Mean  Administrative  Delay  Time  resulted  in 
an  actual  Mean  Maintenance  Down  Time  that  exceeded  the  previously  established  Mean 
Maintenance  Down  Time  goal.  While  Mean  Administrative  Delay  Time  does  not  effect 
other  maintenance  parameters.  Mean  Administrative  Delay  Time  may  affect  the  availabil¬ 
ity  of  a  component  or  system.  To  ensure  availability,  users  should  make  every  effort  to 
keep  Mean  Administrative  Delay  Time  at  a  minimum  by  reporting  nonoperating 
equipment  immediately  following  the  discovery  of  the  equipment’s  nonfunctioning  condi¬ 
tion. 
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TABLE  5.4 

EQUIPMENT  MAINTAINABILITY  GOALS  AND  ACTUAL  VALUES 


PARAMETER 

GOAL 

ACTUAL 

Mean  Time  To  Repair  (MTTR)  On  The  System 

90%  Upper  Limit  To  Repair 

1.5  ManHours 

2.5  ManHours 

0.98  Ma-  Hours 

1 .66  ManHours 

Mean  Time  To  Restore  (MTR) 

90%  Upper  Limit  to  Restore 

0.5  ManHours 

1.0  ManHours 

N/A 

N/A 

(no  data) 

Mean  Time  To  Remove  &  Replace 

90%  Upper  Limit  To  Remove  &  Replace 

0.4  ManHours 

1 .25  ManHours 

0.25  ManHours 

0.66  ManHours 

Direct  Maintenance  Manhours  Per 

Equipment  Operating  Hour 

0.05  Hours 

0.01  Hours 

Response  Time  To  Site  Of  Failure 

4.0  Hours 

0.85  Hours 

Principal  Period  of  Maintenance 

16  Hour/Day 

7  Day/Week 

ik  MMMR— 

Off  Line  Maintenance  Average  MTTR 

90%  Upper  Limit  To  Repair 

1.5  ManHours 

6.0  ManHours 

N/A 

N/A 

(no  data) 

Mean  Preventive  Maintenance  Time 

3.0  Hours 

1.18  Hours 

Mean  Maintenance  Time  (M) 

3.4  Hours 

1 .99  Hours 

Logistic  Delay  Time  (Ldt) 

4.0  Hours 

0.0  Hours 

Maximum  Corrective  Maintenance  Time 

8.0  Hours 

2.0  Hours 

Administrative  Delay  Time  (Aqt) 

2.0  Hours 

16.83  Hours 

Mean  Maintenance  Downtime  (Mqt) 

7.4  Hours 

9.21  Hours 

Maintenance  Manhours  Per  Month 

6.0  Hours/Mo 

2.0  Hours/Mo 

Frequency  of  Preventive  Maintenance 

1  Tlme/Mo  per 
Work  center 

1  Time/Mo  per 

Work  center 

5.1.5  System  Availability 

Table  5.5  displays  Operational  Availability  (A Q)  goals  and  actual  values  for  the  entire 
system  and  three  components  of  the  system:  the  Zenith(TM)248  personal  computer,  the  Z-248 
monitor,  and  the  ALPS^  printer.  In  all  four  instances,  the  actual  operational  availability 
value  met  and  exceeded  the  previously  established  goal.  Actual  operational  availability 
values  of  eight  components  for  which  goal  values  were  unspecified  were  also  calculated. 
The  actual  operational  availability  of  all  eight  components  was  99.64  percent  or  higher. 
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TABLE  5.5 

OPERATIONAL  AVAILABILITY  (A0)  GOALS  AND  ACTUAL  VALUES 


PARAMETER 

SPECIFIED 

ACTUAL 

System  Wide 

95.0% 

99.90% 

8600  Digital  Central  Processing  Unit 

UNSPECIFIED 

99.90% 

56  KBPS  Communication  Lines 

UNSPECIFIED 

99.77% 

992NP  Multiplexer 

UNSPECIFIED 

99.83% 

632  Multiplexer 

UNSPECIFIED 

99.64% 

CSU  Modem 

UNSPECIFIED 

99.96% 

Infotron  Line  Driver 

UNSPECIFIED 

100.00% 

ZWX-248-52  Z-248  Personal  Computer 

96.5% 

99.94% 

ZVM-1380  Z-248  Monitor 

98.0% 

99.97% 

AFP-45  ALPS  Printer  2000 

98.0% 

99.95% 

Scantron  Optical  Mark  Reader 

UNSPECIFIED 

99.96% 

Summagraphics  11x11  Bit  Pad 

UNSPECIFIED 

99.99% 

COMPLIANCE  SUMMARY 

The  prototype  AOTS  met  all  design  and  functional  requirements  specified  for  the  sys¬ 
tem,  subsystem  and  components. 

a.  Appropriate  external  interfaces  were  established. 

b.  Terminal  and  search  response  times  were  conducted  within  the  specified  time 
limits. 

c.  Hardware  reliability  was  at  98.3  percent,  which  exceeded  the  established  god 
of  96.5. 

d.  All  maintenance  goals  were  met  or  exceeded  except  Mean  Administrative 
Delay  Time  (A^t)  and  Mean  Maintenance  Down  Time  (M^t).  The  goals  for 
A^t  and  M^t  were  not  met  due  to  failure  by  users  to  notify  maintenance  activi¬ 
ties  in  a  timely  manner. 

e.  All  hardware  components  were  available  for  use  at  least  99.4  percent  of  the 
total  SLT&E  period. 


5.2  Overall  Results  of  Performance,  Acceptance  and  Suitability  Issues 

The  majority  of  data  used  in  drawing  conclusions  about  the  remaining  critical  issues 
was  collected  via  surveys  of  all  AOTSparticipants  and  interviews  of  those  personnel  found 
to  be  high  frequency  users  of  the  AOTS.  Data  relevant  to  time  spent  performing  training 
responsibilities  before  and  after  the  implementation  of  the  prototype  AOTS  were  also  col¬ 
lected  from  WITS  sheets,  as  discussed  m  Section  4.2.2.  However,  valid  comparisons  of 
time  spent  performing  training  responsibilities  before  and  after  the  AOTS  was 


implemented  could  not  be  made  due  to  the  environmental  impacts.  Factors  such  as  chang¬ 
ing  mission  requirements,  trained  personnel  requirements,  training  opportunities,  and  base 
exercises  and  inspections  dictate  the  time  available  for  training.  Additionally,  these  factors 
differed  significantly  during  the  two  collection  periods. 


The  survey  and  interview  instruments  were  administered  twice,  once  at  the  mid-point 
of  the  SLT&E  period,  and  once  at  the  end  of  the  period.  High  frequency  was  defined  as 
using  AOTS  two  or  more  times  per  week.  Figure  5.1  shows  the  cumulative  survey  and 
interview  results,  by  critical  issue,  for  the  final  issuance. 


TOTAL  SURVEY  RESPONSES  FOR  ALL  USERS 

BY  CRITICAL  ISSUE 

100  -| - 

90  - 
80  - 


47 


47 


FREQUENT  USER  INTERVIEW  RESPONSES 


BY  CRITICAL  ISSUE 


*-34  _ 

W\  POSITIVE  fVx)  NEUTRAL  T777X  NEGATIVE 


Figure  5.1.  Cumulative  Survey  and  Interview  Results 


These  graphs  show  the  total  survey  and  interview  results  for  the  remaining  critical 
issues  Performance,  Acceptance  and  Suitability.  Responses  to  the  surveys  and  structured 
interviews  were  collapsed  into  three  categories:  positive,  neutral  and  negative.  The  bars 
represent  the  percentage  of  responses  that  fell  into  the  three  categories  by  critical  issue. 


As  noted  on  both  graphs,  the  overall  response  to  AOTS  was  positive.  The  high  num 
ber  of  neutrals  was  expected  given  the  brevity  of  the  deployment  period,  limited  use  of  all 
of  the  AOTS  functions,  and  the  specificity  of  the  questions.  However,  a  consistent  test  and 
evaluation  finding  was  that  the  more  time  individuals  spent  learning  the  system,  the  more 
positive  were  their  views  towards  the  AOTS.  In  addition,  higher  frequency  users  were  able 
to  provide  more  valid  positive  and  negative  feedback  on  the  system.  The  subsequent  sec¬ 
tions  of  this  report  will  address  the  test  and  evaluation  findings  of  each  critical  issue  in 
greater  detail. 

5.2.1  Performance  Results 

Unit  level  performance  and  airman  progress  under  AOTS  were  compared  with  per¬ 
formance  and  progress  under  conventional  OJT.  Commanders,  third  party  MAJCOM 
observers  and  a  majority  of  the  AOTS  participants  believed  the  prototype  AOTS  was  an 
improvement  over  conventional  OJT.  This  section  reports  the  individual  results  obtained 
when  analyzing  data  captured  with  each  respective  performance  data  collection  methodol¬ 
ogy.  The  results  of  the  unit  level  data  collection  instruments  are  reported  first,  followed  by 
the  results  obtained  with  airman  level  data  collection  instruments. 
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5.2. 1 . 1  Unit  Level  Performance 

Four  types  of  data  were  collected  to  measure  unit  level  performance:  1)  maintenance 
quality  assurance  technical  inspections  and  personnel  evaluations;  2)  number  of  repeat 
maintenance  actions;  3)  security  police  evaluation  scores;  and  4)  number  of  crime  inci¬ 
dents.  These  existing  operational  measures  were  used  to  determine  the  impact  of  AOTS 
on  conventional  OJT.  However,  the  measures  were  found  not  to  be  an  accurate  reflection 
of  the  OJT  process.  The  limited  availability  (i.e.,  quarterly  reports)  and  recording  bias 
added  to  the  limitations  of  these  measures. 


5.2. 1 .2  Airman  Data  To  Measure  Performance 

Four  types  of  data  were  collected  to  measure  airmen  level  performance:  1)  airman 
training  progress;  2)  proportion  of  time  spent  on  training  management,  training,  and 
administration;  3)  opinions;  and  4)  the  effect  of  administering  computer  based  training 
modules.  The  following  section  reports  the  airman  level  findings  by  data  type. 

5.2. 1.2. 1  Airman  Training  Progress  Data 

Individual  Airman  Training  Record  data  varied  from  month  to  month,  suggesting  that 
either  airmen  changed  duty  positions  or  supervisors  modified  the  number  of  tasks  required 
to  be  position  qualified.  Few  airmen  moved  from  entry-into-a-position  to  full  position 
qualification  during  the  nine  month  interval.  Most  airmen  were  fully  or  partially  position 
qualified  when  the  AOTS  program  began.  Security  police  airmen  were  required  to  become 
folly  position  qualified  within  one  month  of  entry,  regardless  of  whether  trained  under 
AOTS  or  conventional  OJT.  Because  of  the  reactionary  nature  of  OJT  documentation  to 
external  demands  (e.g.,  ORI)  it  was  difficult  to  obtain  valid  rate  of  training  progress  infor¬ 
mation.  In  addition,  some  supervisors  continued  to  change  the  tasks  required  for  dutyposi- 
tions  throughout  the  SLT&E,  making  it  impossible  to  obtain  any  meaningful  results.  Thus, 
data  collected  from  individual  airmen  training  records  became  part  of  the  data  that  were 
excluded  from  analysis. 


5.2. 1.2.2  Time  Spent  on  Training  and  Training  Related  Tasks 

Weekly  Inventory  of  Time  Spent  (WITS)  worksheets  were  completed  by  supervisors, 
training  managers  and  trainees  before  and  during  the  AOTS  SLT&E.  The  worksheets 
were  used  to  track  the  number  of  duty  hours  available,  the  number  of  hours  spent  perform¬ 
ing  training  management  and  administrative  functions,  and  the  number  of  hours  spent 
actually  conducting  or  participating  in  training.  The  intended  purpose  for  collecting  these 
data  was  to  determine  whether  the  proportion  of  time  spent  was  greater  or  less  under  the 
AOTS  than  under  the  conventional  OJT  system.  Comparisons  of  data  collected  before  the 
SLT&E  with  data  collected  during  the  SLT&E  revealed  that:  1)  supervisors  spent  less  than 
one  percent  (.88%)  more  time  under  AOTS  performing  training  management  and  adminis¬ 
trative  functions;  2)  supervisors  and  trainees  spent  less  than  eight  percent  (7.5%)  more 
time  under  AOTS  conducting  and  participating  in  training;  and  3)  training  managers  spent 
almost  six  percent  (5.8%)  less  time  under  AOTS  performing  training  management  func¬ 
tions.  As  previously  discussed  in  Section  5.2,  comparisons  were  considered  invalid  because 
factors  within  the  operational  environment  determined  the  amount  of  time  spent 
performing  training  related  functions,  and  the  impacts  of  those  factors  varied  greatly 
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between  the  two  data  collection  periods.  Therefore,  graphs  displaying  comparisons  of  per¬ 
centages  of  time  spent  performing  task  related  functions  under  the  conventional  system 
with  time  spent  under  AOTS  were  excluded  in  this  report. 


5.2. 1 .3  Performance  Survey  Data 

Performance  survey  items  were  administered  twice  to  supervisors  and  training  man¬ 
agers.  Figure  5.2  displays  the  time  2  data  by  component  and  AFS. 
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Figure  5.2.  Performance  Response  Range  by  Component  and  AFS 


Only  30  supervisors  and  eight  training  managers  responded  twice  to  the  performance 
survey  items.  A  dependent  t  test  comparing  eleven  time  1  and  time  2  ratings  from  the 
guard  component  revealed  a  significant  difference.  The  guard  respondents  gave  a  more 
favorable  rating  at  time  1  than  at  time  2.  A  dependent  t  test  comparing  ten  time  1  and  time 
2  ratings  from  the  security  police  revealed  a  significant  difference.  The  security  police 
respondents  gave  a  more  favorable  rating  at  time  1  than  at  time  2.  The  significant  differ¬ 
ences  between  the  ratings  may  have  been  attributed  to  the  AOTS  being  off  line  in  the 
ANG  security  police  work  center  for  approximately  30  days  between  the  time  1  and  time  2 
administrations  before  system  availability  problems  were  reported  to  the  system  adminis¬ 
trator.  There  were  additional  factors  that  may  have  contributed  to  these  outcomes.  The 
prototype  AOTS  was  designed  to  satisfy  requirements  for  the  majority  of  specialties 
participating  in  the  SLT&E  of  the  system.  Security  Police  policies  established  processes  for 
training,  certification  and  evaluation  requirements  that  differ  from  most  other  mnctional 
areas.  Consequently,  Security  Police  umts  conducted  much  of  their  training  outside  the 
control  of  AOTS.  Also,  due  to  the  nature  of  their  jobs,  Security  Police  members  spent 
most  of  their  duty  time  outside  of  the  work  centers,  where  the  AOTS  work  stations  were 
located  during  the  SLT&E  period. 


Figure  5.3  displays  the  percentage  of  user  responses  falling  in  the  positive,  negative  and 
neutral  ranges.  The  left  bar  graph  displays  the  percentage  of  responses  falling  in  each  cate¬ 
gory  for  all  users  surveyed,  regardless  of  the  amount  of  time  the  user  spent  on  the  system. 
The  right  graph  displays  the  percentage  of  34  AOTS  frequent  user  interview  responses  by 
component. 
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Figure  5.3.  Performance  Response  Range  by  User  Type  and  Component 


Many  users  reported  that  they  believed  individuals  trained  under  the  AOTS  pos¬ 
sessed  more  task  related  knowledge  and  were  better  able  to  perform  tasks  than  airmen 
trained  under  others  methods  of  OJT.  For  the  reasons  discussed  in  Section  5.2. 1.2.1  and 
the  short  SLT&E  period,  it  was  difficult  for  some  users  to  determine  whether  airman 
trained  under  the  prototype  AOTS  performed  better  than  airman  trained  under  conven¬ 
tional  OJT. 


5.2. 1 .4  Performance  Measurement  Study 

In  an  effort  to  take  a  closer  look  at  performance  data,  an  AOTS  job  performance 
study  was  conducted  to  investigate  knowledge  and  performance  differences  attributed  to 
the  use  of  Computer-Based  Training  (CBT).  Airmen  who  received  task  training  via  CBT 
modules  believed  that  the  training  modules  were  beneficial  and  easy  to  use.  Twelve  air¬ 
craft  maintenance  personnel  and  nine  aerospace  propulsion  personnel  selected  for  treat¬ 
ment  groups  viewed  CBT  modules  developed  for  their  respective  specialty.  An  equal 
number  of  control  group  counterparts  did  not  view  the  CBT.  Task  knowledge  tests  and 
over-the-shoulder  perfo  *mance  evaluations  were  administered  to  all  forty-two  airmen.  The 
results  are  depicted  in  Figure  5.4.  Airmen  who  viewed  the  CBT  obtained  significantly 
higher  knowledge  test  scores  than  the  companion  control  group  airmen  who  did  not  review 
the  training  material.  In  addition,  the  treatment  group  personnel  scored  as  well  on  the  per¬ 
formance  evaluations  as  the  control  group  airmen  who  nad  more  experience  performing 
the  respective  tasks. 
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Figure  5.4.  Performance  Measurement  Results 


PERFORMANCE  SUMMARY 


The  analysis  of  those  data  which  were  useful  in  determining  the  performance  of  the 
AOTS  yielded  verypositive  results.  Airmen  trained  using  the  CBT  developed  under  the 
purview  of  the  AOTS  demonstrated  a  higher  level  of  task  knowledge  than  airmen  trained 
under  the  conventional  system,  and  were  equally  qualified  to  perform  tasks,  although  they 
had  considerably  less  experience.  Many  personnel  responding  to  surveys  and  interviews 
indicated  they  believed  the  AOTS  produced  better  qualified  airmen  and  qualified  airmen 
sooner  than  the  conventional  system,  and  that  unit  effectiveness  increasea  under  the 
AOTS. 


Analysis  of  data  collected  to  assess  unit  level  performance  and  airman  training  prog¬ 
ress  revealed  that  comparisons  could  not  be  made.  The  baseline  and  AOTS  data  were  in 
some  instances  inconsistent  or  too  similar  to  determine  which  training  system  performed 
better;  AOTS  or  conventional  OJT. 


5.2.2  Acceptance  Results 

Overall,  respondents  were  positive  about  the  AOTS.  Individuals  familiar  with  the 
AOTS  and  conventional  OJT  methodologies  preferred  the  Advanced  On-the-Job  Training 
System  to  the  conventional  methods.  More  than  three  fourths  (77%)  of  commanders  had  a 
positive  reaction  to  the  AOTS.  Eighty-five  percent  of  commanders  felt  the  AOTS  satisfied 
their  unit  OJT  requirements.  Three  out  of  tour  (75%)  commanders  believed  the  AOTS 
would  significantly  improve  combat  readiness  if  implemented  throughout  the  operational 
environment.  Figure  5.5  displays  the  percentage  of  acceptance  responses  falling  in  the  pos¬ 
itive,  negative  and  neutral  ranges  by  component  and  AFS. 
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Figure  5.5.  Acceptance  Responses  by  Component  and  AFS 


Little  difference  was  noted  across  components.  Individuals  in  Maintenance  and  Per¬ 
sonnel  were  more  favorable  than  raters  from  the  other  specialties.  Security  Police  were 
least  favorable.  As  the  largest  specialties,  maintenance  personnel  liked  the  automated 
functions  for  accessing  and  annotating  training  records.  Personnel  had  a  small  number  of 
individuals  assigned  to  the  specialty  and  most  were  trained  by  the  1ST  or  contractor  train¬ 
ers.  Airmen  assigned  to  Personnel  had  more  experience  with  computers  and  appeared  less 
computer  phobic  than  their  counterparts  in  the  other  specialties.  The  probable  causes  for 
the  lower  ratings  by  the  Security  Police  were  discussed  in  the  Performance  Summary. 


Figure  5.6  displays  the  percentage  of  user  responses  falling  in  the  positive,  negative 
and  neutral  ranges.  The  left  Dar  graph  displays  the  percentage  of  responses  falling  in  each 
category  for  all  users  surveyed,  regardless  of  the  amount  of  time  the  user  spent  on  the  sys¬ 
tem.  The  right  graph  displays  the  percentage  of  34  AOTS  frequent  users’  interview 
responses  f tilling  in  each  response  range  by  component.  A  comparison  of  the  two  graphs 
suggests  that  acceptance  of  the  AOTS  increased  as  use  of  the  system  increased. 
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Figure  5.6.  Acceptance  Responses  by  User  type  and  Component 
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More  than  one-third  of  all  individuals  responded  in  the  neutral  acceptance  range. 
Not  all  personnel  were  able  to  distinguish  between  training  and  task  performance  because 
much  of  conventional  on-the-job  training  was  conducted  during  the  performance  of  opera¬ 
tional  tasks.  Some  individuals  had  an  insufficient  understanding  of  Air  Force  OJT  and 
required  an  orientation  on  generic  OJT  practices  before  attempting  to  learn  the  AOTS 
functions.  One  hundred  percent  of  all  AOTS  participants’  acceptance  mean  ratings  fell  in 
the  positive  or  neutral  categories  indicating  overall  user  acceptance  with  the  AOTS. 

Evaluation  of  the  AOTS  by  eighty-two  third  party  MAJCOM  observers  after  a  two 
day  orientation  supplemented  the  evaluation  efforts.  These  individuals  evaluated  AOTS 
from  a  perspective  of  conceptually  using  the  system  at  both  the  command  and  AF-wide 
level.  While  the  observers  were  familiar  with  the  deficiencies  in  AF  OJT,  they  were  not 
bound  by  the  constraints  of  the  operational  setting  in  evaluating  the  system.  Figure  5.7 
shows  the  percentage  of  positive,  negative  and  neutral  responses  by  observer.  Sixty-four 
percent  of  the  observers  liked  the  AOTS  and  believed  the  system  was  easy  to  use. 
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Figure  5.7.  Percentage  of  Observer  Responses  in  Each  Acceptance  Range 
ACCEPTANCE  SUMMARY 


The  overall  results  for  the  Acceptance  critical  issue  revealed  that  the  respondents 
liked  the  prototype  AOTS  and  felt  it  was  easy  to  use.  This  finding  was  consistent  across  all 
components,  ana  with  the  exception  of  the  Security  Police,  across  all  AFSs.  There  were 
communication  line  problems  and  unique  training  process  differences  among  the  Security 
Police  that  may  have  contributed  to  this  finding.  MAJCOM  observers  and  higher  fre¬ 
quency  users  of  the  system  reported  the  most  positive  view  of  the  system. 


5.2.3  Suitability  Results 

According  to  the  majority  of  the  respondents,  the  AOTS  was  viewed  as  overcoming 
identified  OJT  deficiencies  and  should  be  implemented.  For  example,  seventy-three  per¬ 
cent  of  all  AOTS  participants  responding  to  the  opinion  survey  believed  a  system  similar  to 
the  AOTS  is  needed  in  tneir  career  fields.  Fifty-seven  percent  favored  the  AOTS  Airman 
Training  Record  over  AF  Form  623  for  documenting  OJT  data.  Fifty-two  percent  believed 
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the  AOTS  should  be  implemented  in  the  operational  environment  AF  wide.  Almost  all 
commanders  (92%)  believed  the  AOTS  was  suitable  and  should  be  implemented  across  the 
Air  Force.  Figure  5.8  shows  the  percentage  of  supervisor,  training  manager  and  trainer/e¬ 
valuator  responses  falling  in  the  positive,  negative  and  neutral  ranges  by  component  and 
AFS. 
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Figure  5.8.  Suitability  Responses  by  Component  and  AFS 


The  guard  respondents  gave  a  higher  suitability  rating  at  time  1  than  at  time  2.  A 
dependent  t  test  comparing  10  time  1  and  time  2  ratings  from  the  security  police  also 
revealed  significantly  higher  suitability  ratings  at  time  1  than  at  time  2.  As  with  the  Secu¬ 
rity  Police  acceptance  ratings,  the  lower  Security  Police  suitability  ratings  at  time  2  may 
have  been  due  to  equipment  and  communication  line  problems  and  unique  training 
differences. 


Figure  5.9  displays  the  percentage  of  user  responses  falling  in  the  positive,  negative 
and  neutral  ranges.  The  left  bar  graph  displays  the  percentage  or  responses  falling  in  each 
category  for  all  users  surveyed,  regardless  of  the  amount  of  time  the  user  spent  on  the  sys¬ 
tem.  The  right  graph  displays  the  percentage  of  34  AOTS  frequent  users’  interview 
responses  falling  in  each  category  by  component.  Other  suitability  questions  revealed  that 
sixty-five  percent  of  the  supervisors  believed  that  their  career  field  needed  an  on-the-job 
training  system  similar  to  the  AOTS.  Sixty-one  percent  indicated  liking  one  or  more  of  the 
AOTS  features.  Twenty-two  percent  liked  using  the  entire  AOTS  system.  Forty-seven  per¬ 
cent  of  the  supervisors  believed  that  AOTS  should  be  implemented  in  the  Air  Force. 
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Figure  5.9.  Suitability  Response  Range  by  User  Type  and  Component 


The  majority  of  training  managers  (77%)  preferred  the  prototype  AOTS  to  the  con¬ 
ventional  OJT  system.  More  than  half  (53%)  indicated  they  favored  the  AOTS  automated 
methodologies  for  recording  training  requirements  and  history  over  maintaining  manual 
AF  Forms  623.  Three  out  of  four  (75%)  of  the  training  managers  believed  that  their  career 
field  would  benefit  greatly  through  implementation  of  an  on-the-job  training  system  similar 
to  the  AOTS.  Fifty  percent  of  the  training  managers  believed  AOTS  would  significantly 
increase  combat  capabilities.  A  significant  number  (63%)  believed  that  the  application  of 
training  technologies  demonstrated  via  the  AOTS  would  enable  trainees  to  complete  train¬ 
ing  requirements  more  efficiently.  An  even  greater  number  (65%)  of  the  training  manag¬ 
ers  believed  that  AOTS  should  be  implemented  throughout  the  Air  Force. 


After  completing  a  two-day  orientation,  the  majority  (77%)  of  MAJCOM  observers 
believed  they  had  received  sufficient  information  to  adequately  assess  the  potential  of 
AOTS.  Almost  nine  out  of  ten  (87%)  indicated  their  behef  that  the  conventional  OJT  sys¬ 
tem  has  severe  deficiencies  and  needs  to  be  improved.  Analysis  of  responses  to  a  survey 
having  questions  specifically  related  to  the  suitability  issue  revealed  that  seventy-one 
percent  of  the  MAJCOM  observers  believed  the  prototype  AOTS  met  or  exceeded  Air 
Force  OJT  policies,  as  documented  in  AFR  50-23.  Nearly  3  out  of  4  (74%)  of  the  MAJ¬ 
COM  observers  believed  their  functional  area  or  MAJCOM  had  a  need  for  an  OJT  system 
similar  to  AOTS,  and  more  than  half  (54%)  believed  AOTS  could  improve  the  combat 
readiness  status  of  MAJCOMs.  More  than  three  fourths  of  the  observers  (78%)  believed 
all  OJT  systems  operating  in  their  functional  areas  required  management,  training  devel¬ 
opment  and  delivery,  ana  evaluation  development  and  delivery  functions.  Figure  5.10  dis¬ 
plays  the  percentage  of  MAJCOM  observer  responses  falling  in  the  positive,  negative  and 
neutral  response  ranges.  The  majority  (74%)  of  the  third  party  observers’  responses  were 
in  the  positive  range. 
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Figure  5.10.  Percentage  of  Observer  Responses  in  Each  Suitability  Range 
SUITABILITY  SUMMARY 

Overall  results  for  the  Suitability  critical  issue  revealed  that  the  prototype  AOTS 
solved  existing  OJT  deficiencies  and  that  the  system  functioned  well  within  the  operational 
environment.  This  finding  was  consistent  across  all  components  and  AFSs.  The  only 
exception  was  Security  Police.  Respondents  who  were  most  familiar  with  Air  Force  OJT 
policies  and  procedures  and  with  the  AOTS  responded  more  favorably.  These  respondents 
also  believed  the  AOTS  is  needed  in  the  operational  environment  and  should  be  implem¬ 
ented  Air  Force  wide. 


5.3  Process  Data  Results 

AOTS  training  was  delivered  independently  to  each  user  to  allow  the  level  and  pace 
of  the  training  to  be  tailored  for  each  individual.  Contractors  and  members  of  the  Air 
Force  Instructional  Systems  Team  (1ST)  trained  supervisors,  training  managers,  and  quality 
control  monitors  to  use  AOTS.  The  responsibility  of  training  trainers,  evaluators  and  train¬ 
ees  fell  on  the  trained  supervisors  and,  in  some  instances,  training  managers.  After  the  ini¬ 
tial  training,  many  individuals  became  more  proficient  on  the  system.  However,  some 
management  personnel  reported  feeling  overwhelmed  by  the  initial  instruction  on  the 
system  and  avoided  seeking  further  training.  Lack  of  motivation  by  the  supervisors  to  learn 
AOTS  and  instruct  their  subordinates  required  the  subordinate  personnel  to  learn  AOTS 
on  their  own  or  to  request  training  from  an  1ST  member.  Supplemental  training  was  avail¬ 
able  to  anyone  seeking  additional  training.  Many  of  the  trainers  and  trainees  who  received 
inadequate  training  remained  infrequent  users  of  the  AOTS  and  indicated  on  their  opinion 
surveys  that  they  disliked  or  were  neutral  toward  the  prototype  AOTS. 

A  "hot-line"  for  reporting  problems  and  requesting  additional  training  was  available 
for  all  users  24  hours  a  day.  Novice  users  reported  simple  problems  (e.g.,  logging  onto  the 
system).  Frequent  users  reported  more  complicated  problems  and  required  assistance  with 
complex  functions  such  as  scheduling  training  for  one  or  more  trainees. 
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AOTS  log-on  records  and  observation  reports  revealed  that  users  who  learned  to  use 
the  system  liked  the  AOTS  automated  functions,  and  used  the  functions  to  complete  their 
OJT  responsibilities.  Usage  patterns  were  examined  by  components  (i.e..  Active,  Air 
National  Guard  and  Air  Force  Reserves),  specialties  (i.e..  An  craft  Maintenance,  Aero¬ 
space  Propulsion,  Security  Police  and  Personnel)  and  preferred/least  preferred  AOTS 
function  (e.g.,  Individual  Airman  Training  Record,  Quality  Control,  On-line/Off-line 
evaluations). 


After  the  initial  two  months  of  training  on  the  AOTS,  there  was  a  gradual  increase  in 
usage  among  the  Air  Force  Active  Duty  personnel  and  a  noticeable  drop  in  usage  by  Air 
National  Guard  and  Air  Force  Reserve  personnel.  Air  Force  Active  Duty  personnel  con¬ 
tinued  to  show  an  increase  in  usage  until  the  months  of  January  and  February.  In  contrast, 
the  Air  National  Guard  and  Air  Force  Reserve  personnel  maintained  a  low  and  stable 
pattern  of  usage  during  most  of  deployment  penod  except  for  a  peak  usage  period  during 
January  for  the  Air  National  Guard. 


The  differential  usage  pattern  among  components  may  have  been  a  function  of  differ¬ 
ences  in  mission  requirements  and  field  activities.  For  example,  Air  Force  Active  Duty 
personnel  were  able  to  utilize  the  system  on  a  full  time  basis.  With  the  limited  exception  of 
full-time  personnel,  Air  National  Guard  and  Air  Force  Reserve  unit  personnel  used  the 
system  only  one  weekend  per  month,  during  Unit  Training  Assembly. 

There  was  substantial  use  of  the  AOTS  functions  to  review  OJT  records  and  update 
task  certification  and  position  qualification  status  during  February.  The  peak  period  of  use 
among  the  Air  Force  Active  Duty  personnel  in  January  and  February  and  the  gradual 
decline  after  February  may  have  been  the  results  of  several  base  exercises  leading  to  an 
Operational  Readiness  Inspection  (ORI)  in  February. 

Among  the  Air  Force  Active  Duty  personnel,  the  Aircraft  Maintenance  personnel 
used  the  system  most  often.  Aircraft  Maintenance  had  the  largest  number  of  personnel; 
consequently,  it  is  not  surprising  that  they  would  have  the  highest  usage  on  the  system. 
However,  Aircraft  Maintenance  use  after  February  fell  to  a  level  below  the  usage  rate  of 
the  Security  Police  specialty  for  the  remaining  months  of  the  deployment  period. 


Within  the  Air  Force  Reserves,  system  utilization  among  the  Aircraft  Maintenance 
and  Aerospace  Propulsion  specialties  exceeded  the  usage  by  Personnel  and  Security  Police 
specialties.  This  finding  was  expected  given  the  disproportionately  higher  number  of  per¬ 
sonnel  in  the  two  maintenance  specialties.  However,  among  the  Air  National  Guard  spe¬ 
cialties,  the  Security  Police  exhibited  the  greatest  amount  of  usage,  except  for  the  months 
of  April  and  June,  when  the  Aircraft  Maintenance  specialty  used  the  system  more. 

The  Airman  Training  Record  (ATR)  was  the  most  frequently  accessed  data  base. 
The  capabilities  in  the  management  subsystem  to  access  and  update  training  records  such 
as  identifying  training  requirements,  scheduling  training  events  and  certifying  task  qualifi¬ 
cations  were  used  more  frequently  than  those  in  the  evaluation  or  training  development 
and  delivery  subsystems. 
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6  DISCUSSION 


The  primary  objective  of  this  project  was  to  determine  if  deficiencies  in  the  conven¬ 
tional  Air  Force  enlisted  OJT  system  could  be  resolved  through  applying  modem  computer 
and  training  technologies.  Lower  level  objectives  were  established  for  testing  and 
evaluating  the  system  so  that  a  determination  could  be  made  as  to  whether  the  primary 
objective  was  met.  These  lower  level  objectives  were  appropriately  aligned  with  the  critical 
issues  Compliance,  Performance,  Acceptance  and  Suitability. 


Two  types  of  testing  and  evaluation  occurred.  First,  testing  of  the  system  occurred 
during  development  to  determine  whether  all  functions  specified  were  included,  whether 
the  functions  performed  as  specified,  and  whether  the  system  was  ready  to  be  installed  in 
the  participating  operational  work  centers.  The  results  of  this  testing  were  used  to  measure 
Compliance.  Second,  following  development  and  installation,  the  system  was  tested  and 
evaluated  to  determine  the  impacts  it  had  on  work  center  training  programs  and  how  it  was 
perceived  by  users  and  MAJCOM  observers.  The  results  of  this  testing  and  evaluation 
were  used  to  measure  Performance,  Acceptance  and  Suitability. 


A  twelve-month  period  was  scheduled  for  the  SLT&E  of  the  prototype  AOTS.  How¬ 
ever,  thisperiod  was  considered  too  brief  to  effectively  compare  the  AOTS  to  the  conven¬ 
tional  OJT  system.  At  least  four  factors  served  as  indicators  that  more  time  should  have 
been  allotted  for  the  SLT&E.  These  were  1)  the  time  required  for  users  to  learn  to 
operate  the  system,  2)  the  time  required  to  teach  work  center  personnel  Air  Force  OJT 
policies  and  procedures,  3)  the  infrequent  use  of  the  system  by  many  work  center  person¬ 
nel,  and  4)  the  need  to  terminate  the  collection  of  data  two  months  prior  to  the  end  of  the 
period  to  enable  analyses  to  occur.  Although  the  AOTS  could  not  be  effectively  compared 
with  the  conventional  system,  results  of  assessments  by  users  and  MAJCOM  observers 
clearly  indicated  whether  established  objectives  were  met.  Discussions  relevant  to  the 
results  of  analyzing  data  collected  before  and  after  the  implementation  of  the  prototype 
AOTS  are  presented  below,  by  critical  issue. 

COMPLIANCE 

The  prototype  AOTS  met  all  requirements  specified  for  the  system,  subsystems  and 
components. 

a.  Appropriate  external  interfaces  were  established; 

b.  Terminal  and  search  response  times  were  conducted  within  specified  time  lim¬ 
its; 

c.  Hardware  reliability  exceeded  the  established  goal; 

d.  All  maintenance  goals  were  met  or  exceeded  except  in  instances  where  users 
failed  to  report  problems; 

e.  All  hardware  component  availability  exceeded  established  goals;  and 

f.  The  system  was  available  seven  days  a  week,  to  all  shifts  and  all  participating 
work  centers. 


J 
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PERFORMANCE 

The  analysis  of  position  qualification  data  collected  to  compare  airmen  trained  under 
the  AOTS  with  airmen  trained  under  the  conventional  system  revealed  that  valid  compari¬ 
sons  could  not  be  made.  TTie  majority  of  airmen  were  position  qualified  before  data  collec¬ 
tion  began.  For  those  airmen  who  were  actively  participating  in  training,  the  number  of 
tasks  required  for  position  qualification  did  not  remain  constant  from  month  to  month. 
Airmen  were  frequently  reassigned  to  different  duty  positions.  Also,  few  airmen  trained 
under  AOTS  entered  position  qualification  training  and  became  position  qualified  during 
the  SLT&E  period. 

Analysis  of  data  collected  via  WITS  sheets  revealed  that  comparisons  of  time  spent 
performing  training  responsibilities  before  the  implementation  of  the  prototype  AOTS  with 
time  spent  after  implementation  were  difficult  to  make,  due  to  environmental  impacts. 

Analysis  of  performance  data  revealed  that  maintenance  personnel  trained  under  the 
AOTS  consistently  scored  higher  on  knowledge  tests  than  did  personnel  trained  under  the 
conventional  system.  When  given  performance  tests,  personnel  trained  under  the  AOTS 
scored  as  well  as  personnel  trained  under  the  conventional  system,  even  though  these  air¬ 
men  had  significantly  less  experience  in  performing  the  tasks. 

The  analysis  of  survey  and  interview  data  revealed  that  users  of  the  prototype  AOTS 
believed  that  airmen  trained  under  the  system  achieved  task  qualification  sooner,  and  were 
better  qualified  to  perform  tasks  than  were  airmen  trained  under  the  conventional  OJT  sys¬ 
tem.  Also,  the  majority  of  personnel  surveyed  and  interviewed  responded  positively  when 
asked  if  the  prototype  AOTS  improved  performance  of  the  overall  OJT  system,  saved  time 
during  the  performance  of  training  responsibilities,  improved  unit  effectiveness,  and  would 
improve  combat  readiness  if  implemented  throughout  the  operational  environment. 


The  analyses  of  unit  assessment  data  (Aircraft  Maintenance  quality  assurance  and 
repeat  maintenance  statistics,  and  Security  Police  quality  control  and  base  crime  statistics) 
showed  there  were  no  significant  differences  between  baseline  and  AOTS  data. 

ACCEPTANCE 

The  results  from  analyzing  survey  and  interview  data  clearly  indicated  a  high  level  of 
acceptance  of  the  prototype  AOTS  by  frequent  users  of  the  system.  Users  favored  the 
methods  for  managing,  scheduling  and  documenting  training  provided  with  the  AOTS  over 
conventional  methods. 
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SUITABILITY 

The  analysis  of  survey  and  interview  data  showed  that  most  users  and  MAJCOM 
observers  believed  the  AOTS  was  needed  to  enhance  training  within  their  respective  career 
fields.  Further,  these  personnel  believed  the  AOTS  was  suitable  for  deployment  through¬ 
out  the  Air  Force  operational  environment,  and  should  be  implemented.  Unit  command¬ 
ers  who  were  interviewed  responded  very  positively  about  the  prototype  AOTS.  These 
commanders  believed  the  AOTS  satisfied  their  umt  OJT  requirements,  would  significantly 
improve  the  combat  capabilities  of  their  units,  was  suitable  for  Air  Force-wide  use,  and 
should  be  implemented. 
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7  CONCLUSION  AND  RECOMMENDATION 


The  execution  of  the  prototype  Advanced  On-the-Job  Training  System  at  Bergstrom 
Air  Force  Base  and  Ellington  Air  National  Guard  Base  was  successful.  The  project  was 
completed  on  schedule  and  within  its  budget.  The  experiment  proved  that  a  total  training 
system  as  defined  by  AFHRL  could  be  developed  and  implemented  in  the  operational 
environment.  The  AOTS  successfully  provided  AFHRL  specified  on-line  capabilities  and 
methodologies  to: 

a.  Identify  and  define  task  performance  and  training  requirements; 

b.  Develop  and  deliver  training  and  evaluation  materials; 

c.  Schedule  training  and  evaluation  events,  and  identify  performance,  training 
and  evaluation  resource  requirements; 

d.  Track  and  record  individual  airman  training  accomplishments; 

e.  Evaluate  individual  airman  task  knowledge  and  performance; 

f.  Track,  evaluate,  and  report  the  AOTS  program  effectiveness;  and 

g.  Report  airman  training,  evaluation  status,  and  evaluation  results  to  appropri¬ 
ate  Air  Force  managers. 


The  AOTS  met  performance  and  hardware  reliability,  maintainability,  and  availabil¬ 
ity  goals  and  provided  the  required  interfaces  with  appropriate  external  entities.  During 
the  implementation  of  the  AOTS,  the  system  was  available  99  percent  of  the  time.  The 
amount  of  time  required  for  maintenance  personnel  to  recover  from  an  error  and  bring  the 
AOTS  or  its  components  back  to  an  operational  mode  was  minimal.  The  probability  of  the 
AOTS  operating  successfully  in  the  operational  environment  was  significantly  high.  Indi¬ 
viduals  familiar  with  the  AOTS  capabilities  and  OJT  methodologies  expressed  a  desire  to 
deploy  the  AOTS  in  the  operational  environment.  After  assessing  the  prototype  AOTS,  a 
committee  of  Air  Staff  and  MAJCOM  functional  management  representatives  agreed  that 
a  system  such  as  the  prototype  AOTS  was  needed  to  provide  standardized  training  and 
evaluation  procedures  Air  Force  wide. 


As  recommended  by  the  commanders,  observers  and  frequent  users  of  the  AOTS,  the 
system  should  be  implemented  in  the  operational  environment  Air  Force  wide  to  help 
increase  the  quality  of  training  programs  within  the  Air  Force  organizations. 


8  ABBREVIATIONS  AND  ACRONYMS 


Adt 

Administrative  Delay  Time 

AFB 

Air  Force  Base 

AFS 

Air  Force  Specialty 

AFSC 

Air  Force  Specialty  Code 

AMU 

Aircraft  Maintenance  Unit 

ANG 

Air  National  Guard 

ANGB 

Air  National  Guard  Base 

ANOVA 

Analysis  of  Variance 

Ao 

Operational  Availability 

AOTS 

Advanced  On-the-job  Training  System 

ATC 

Air  Training  Command 

CAM 

Criteria  Acquisition  Model 

HQ  USAF/DPP  Headquarters  United  States  Air  Force  /Director  of  Personnel 
grams 

1ST 

Instructional  System  Team 

JEIM 

Jet  Engine  Intermediate  Maintenance 

Ldt 

Logistic  Delay  Time 

M 

Mean  Maintenance  Time 

MAC 

Military  Airlift  Command 

MAJCOM 

Major  Command 

MDC 

Mean  Data  Collection 

MILAP 

Maintenance  Information  Logically  Analyzed  and  Presented 

Mdt 

Mean  Maintenance  Downtime 

MTBF 

Mean  Time  Between  Failures 

MTL 

Master  Task  List 

MTP 

Master  Test  Plan 

MTR 

Mean  Time  To  Restore 

MTTR 

Mean  Time  To  Repair 

NCO 

Noncommissioned  Officer 

OJT 

On-the-Job  Training 

ORI 

Operational  Readiness  Inspection 

QA 

Quality  Assurance 

QC 

Quality  Control 

SAC 

Strategic  Air  Command 
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SLT&E 

SME 

SOA 

SPOL 

TAC 

UTA 

WITS 


System  Level  Test  &  Evaluation 

Subject  Matter  Expert 

Separate  Operating  Agency 

Security  Police  Operations-Law  Enforcement 

Tactical  Air  Command 

Unit  Training  Assembly 

Weekly  Inventory  of  Time  Spent 
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